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Abstract 


One  of  the  most  challenging  activities  for  a  distance  education  facilitator  is  to  be  responsive  to 
student  needs  and  customize  the  learning  experience  to  the  student’s  individual  learning  style, 
help  socialize  the  student  to  the  institution,  and  keep  the  learner  engaged  in  spite  of  the  isolated 
environment.  To  achieve  these  goals,  mechanisms  to  improve  efficiency  and  effectiveness  for  the 
Canadian  Forces  (CF)  distance  learning  capability  need  to  be  explored.  To  support  these  goals,  a 
Requirement  and  Stakeholder  Analysis  for  the  CF  Counter-IED  Training  Courses  was 
undertaken  to  investigate  the  requirements  of  stakeholders  of  the  CF  IED  Disposal  (IEDD) 
Operator  Training  Course  for  the  development  of  adaptive  learning  technology  integration  and 
validation  plans.  Given  the  high  failure  rate  in  the  existing  IEDD  Operator  Course,  problematic 
aspects  of  the  course  was  one  of  the  foci  of  the  Stakeholder  Analysis.  The  discussion  with 
stakeholders  focused,  in  part,  on  the  feasibility  of  demonstrating  the  capability  of  intelligent 
tutoring  technology  within  the  IEDD  Operator  Course.  The  report  presents  the  results  of  the 
Stakeholder  Analysis  interviews  and  recommends  the  implementation  plan  for  the  IEDD 
Operator  Course  Intelligent  Tutoring  System  (ITS),  in  terms  of  its  content  and  capabilities,  and 
describes  the  notional  architecture  of  the  IEDD  ITS  in  more  detail.  The  report  also  recommends 
the  evaluation  plan  for  the  IEDD  Operator  Course  ITS  in  terms  of  the  methodology  and 
approximate  schedule  for  activities  pertaining  to  the  evaluation  of  the  ITS. 


Resume 


L’une  des  taches  les  plus  difficiles  qui  incombe  a  l’instructeur  a  distance  est  de  repondre  aux 
besoins  de  l’apprenant  et  de  personnaliser  l’acquisition  de  connaissances  en  fonction  du  style 
d’apprentissage  du  stagiaire,  d’entretenir  la  relation  entre  le  stagiaire  et  l’etablissement  de 
fonnation  et  de  conserver  la  motivation  de  l’apprenant  malgre  son  isolement.  Pour  y  parvenir,  il 
faut  examiner  des  moyens  d’ameliorer  la  rentabilite  et  l’efficacite  de  la  capacite  d’apprentissage 
a  distance  des  Forces  canadiennes  (FC).  Dans  cette  optique,  on  a  entrepris  une  Analyse  des 
besoins  et  des  intervenants  des  cours  d  ’instruction  des  FC  sur  la  neutralisation  des  IED  a  fin  de 
connaitre  les  besoins  des  intervenants  du  cours  d’operateur  de  neutralisation  des  engins  explosifs 
improvises  (IEDD)  des  FC  en  matiere  d’ integration  et  de  validation  de  technologies 
d’apprentissage  adaptatif.  Vu  le  taux  eleve  d’echec  au  cours  actuel  d’operateur  IEDD,  les  aspects 
problematiques  du  cours  ont  ete  cibles  durant  1 ’Analyse  des  intervenants.  Les  echanges  avec  les 
intervenants  ont  porte  entre  autres  sur  la  faisabilite  de  la  demonstration  de  la  capacite  de  la 
technologie  des  tutoriels  intelligents  dans  le  cadre  du  cours  d’operateur  IEDD.  Le  rapport 
presente  les  resultats  des  entrevues  realisees  aux  fins  de  l’Analyse  des  intervenants  et 
recommande  le  plan  de  mise  en  oeuvre  du  systeme  tutoriel  intelligent  (ITS)  du  cours  d’operateur 
IEDD,  plus  precisement  son  contenu  et  ses  capacites,  et  explique  plus  en  detail  1’ architecture 
theorique  du  tutoriel  IEDD.  On  recommande  aussi  dans  le  rapport  le  plan  devaluation  du 
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systeme  tutoriel  intelligent  du  cours  d’operateur  IEDD,  des  points  de  vue  de  la  methodologie  et 
du  calendrier  approximatif  des  activites  touchant  1’ evaluation  du  ITS. 
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Executive  summary 


Stakeholder  Analysis  for  the  CF  Counter-IED  Training  Course 

Simon  Branbury,  Kristina  Osgoode,  David  Unrau,  Chelsea  Kramer,  Jordan  Miller; 
DRDC  Toronto  CR  2010-059;  Defence  R&D  Canada  -  Toronto. 

This  document  presents  the  results  of  a  Stakeholder  Analysis  of  Canadian  Forces  (CF)  Counter 
Improvised  Explosive  Device  (IED)  training  courses.  The  Stakeholder  Analysis  was  used  to 
inform  the  design,  development  and  evaluation  of  an  Intelligent  Tutoring  System  (ITS)  for  the 
IED  Disposal  (IEDD)  Operator  Course.  This  work  was  conducted  under  Contract  W771 1-09- 
8153/001/TOR  “Intelligent  Tutoring  for  Distance  Learning”  for  Defence  Research  Development 
Canada  (DRDC)  Toronto  by  CAE  Professional  Services  (CAE  PS). 

One  of  the  most  challenging  activities  for  a  distance  education  facilitator  is  to  be  responsive  to 
student  needs  and  customize  the  learning  experience  to  the  student’s  individual  learning  style, 
help  socialize  the  student  to  the  institution,  and  keep  the  learner  engaged  in  spite  of  the  isolated 
environment.  To  achieve  these  goals,  Defence  Research  and  Development  Canada  (DRDC)  - 
Toronto  has  initiated  an  Applied  Research  Project  (ARP)  to  investigate  mechanisms  to  improve 
efficiency  and  effectiveness  for  the  Canadian  Forces  (CF)  distance  learning  capability. 

In  order  to  support  these  goals,  a  Requirement  and  Stakeholder  Analysis  for  the  CF  Counter-IED 
Training  Courses  was  undertaken  by  CAE  PS  to  investigate  the  requirements  of  stakeholders  of 
the  CF  IEDD  Operator  Course  for  the  development  of  adaptive  learning  technology  integration 
and  validation  plans.  Given  the  high  failure  rate  in  the  existing  IEDD  Operator  Course, 
problematic  aspects  of  the  course  was  one  of  the  foci  of  the  Stakeholder  Analysis.  The  discussion 
with  stakeholders  focused,  in  part,  on  the  feasibility  of  demonstrating  the  capability  of  intelligent 
tutoring  technology  within  the  IEDD  Operator  Course. 

The  report  presents  the  results  of  the  Stakeholder  Analysis  interviews  conducted  between  the 
22nd  January  2010  and  the  23rd  February  2010,  and  describes  the  recommendations  for 
implementing  adaptive  learning  and  intelligent  tutoring  technologies  into  a  suitable  CF  distance¬ 
learning  course.  Following  this,  the  report  describes  the  implementation  plan  for  the  IEDD 
Operator  Course  ITS,  in  terms  of  its  content  and  capabilities,  and  describes  the  notional 
architecture  of  the  IEDD  ITS  in  more  detail,  as  well  as  the  schedule  for  the  design, 
implementation  and  evaluation  of  the  ITS.  Finally,  the  report  describes  the  evaluation  plan  for 
the  IEDD  Operator  Course  ITS  in  terms  of  the  methodology  and  approximate  schedule  for 
activities  pertaining  to  the  evaluation  of  the  ITS. 

The  report  also  recommends  a  number  of  tasks  to  be  conducted  early  during  the  next  phase  of  the 
project  in  order  to  provide  more  detailed  plans  for  the  implementation  of  ITS  technologies  within 
the  IEDD  Operator  Course: 

•  Questioning  technique  literature  review; 
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•  Development  and  validation  of  the  IEDD  ITS  mission  scenario; 

•  Conduct  taxonomic  analysis  (including  instructional  design);  and, 

•  IEDD  ITS  development  (iteration  1). 
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Sommaire 


Analyse  des  intervenants  des  cours  d’instruction  des  FC  sur  la 
neutralisation  des  IED 

Simon  Branbury,  Kristina  Osgoode,  David  Unrau,  Chelsea  Kramer,  Jordan  Miller; 

DRDC  Toronto  CR  2010-059;  Defence  R&D  Canada  -  Toronto 

L’une  des  taches  les  plus  difficiles  qui  incombe  a  l’instructeur  a  distance  est  de  repondre  aux 
besoins  de  l’apprenant  et  de  personnaliser  [’acquisition  de  connaissances  en  fonction  du  style 
d’apprentissage  du  stagiaire,  d’entretenir  la  relation  entre  le  stagiaire  et  l’etablissement  de 
formation  et  de  conserver  la  motivation  de  l’apprenant  malgre  son  isolement.  Pour  y  parvenir, 
Recherche  et  developpement  pour  la  defense  Canada  (RDDC)  -  Toronto  a  lance  un  Projet  de 
recherche  appliquee  (PRA)  afin  d’examiner  des  moyens  d’ameliorer  la  rentabilite  et  l’efficacite 
de  la  capacite  d’apprentissage  a  distance  des  Forces  canadiennes  (FC). 

Dans  cette  optique,  on  a  entrepris  une  Analyse  des  besoins  et  des  intervenants  des  cours 
d’instruction  des  FC  sur  la  neutralisation  des  IED  afin  de  connaitre  les  besoins  des  intervenants 
du  cours  d’operateur  de  neutralisation  des  engins  explosifs  improvises  (IEDD)  des  FC  en  matiere 
d’integration  et  de  validation  de  technologies  d’apprentissage  adaptatif.  Vu  le  taux  eleve  d’echec 
au  cours  actuel  d’operateur  IEDD,  les  aspects  problematiques  du  cours  ont  ete  cibles  durant 
l’Analyse  des  intervenants.  Les  echanges  avec  les  intervenants  ont  porte  entre  autres  sur  la 
faisabilite  de  la  demonstration  de  la  capacite  de  la  technologie  des  tutoriels  intelligents  dans  le 
cadre  du  cours  d’operateur  IEDD. 

Le  rapport  presente  les  resultats  des  entrevues  realisees  aux  fins  de  l’Analyse  des  intervenants  et 
enonce  des  recommandations  pour  la  mise  en  oeuvre  de  technologies  d’apprentissage  adaptatif  et 
de  tutoriels  intelligents  dans  le  cadre  d’un  cours  pertinent  d’apprentissage  a  distance  des  FC. 
Ensuite,  le  rapport  decrit  le  plan  de  mise  en  oeuvre  du  systeme  tutoriel  intelligent  (ITS)  du  cours 
d’operateur  IEDD,  plus  precisement  son  contenu  et  ses  capacites,  et  explique  plus  en  detail 
1’ architecture  theorique  du  tutoriel  IEDD,  ainsi  que  le  calendrier  de  conception,  de  mise  en 
oeuvre  et  devaluation  du  ITS.  En  conclusion,  on  trouve  le  plan  devaluation  du  systeme  tutoriel 
intelligent  du  cours  d’operateur  IEDD,  des  points  de  vue  de  la  methodologie  et  du  calendrier 
approximate  des  activites  touchant  l’evaluation  du  ITS. 

On  recommande  aussi  dans  le  rapport  des  taches  a  accomplir  au  debut  de  la  prochaine  etape  du 
projet  afin  d’en  arriver  a  des  plans  plus  detailles  pour  la  mise  en  oeuvre  de  technologies  de 
tutoriels  intelligents  dans  le  cadre  du  cours  d’operateur  IEDD  : 

•  examen  de  la  documentation  concemant  les  techniques  d’ interrogation; 

•  elaboration  et  validation  du  scenario  de  mission  en  vue  du  systeme  tutoriel  intelligent 
(ITS)  du  cours  d’operateur  IEDD; 
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•  analyse  taxinomique  (y  compris  la  conception  pedagogique); 
developpement  du  systeme  tutoriel  intelligent  en  matiere  d’lEDD  (premiere  iteration). 
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1  INTRODUCTION 

This  document  presents  the  results  of  a  Stakeholder  Analysis  of  Canadian  Forces  (CF) 
Counter  Improvised  Explosive  Device  (IED)  training  courses.  The  Stakeholder  Analysis 
was  used  to  inform  the  design,  development  and  evaluation  of  an  Intelligent  Tutoring 
System  (ITS)  for  the  IED  Disposal  (IEDD)  Operator  Course.  This  work  was  conducted 
under  Contract  W771 1-09-81 53/001 /TOR  “Intelligent  Tutoring  for  Distance  Learning”  for 
Defence  Research  Development  Canada  (DRDC)  Toronto  by  CAE  Professional 
Services  (CAE  PS). 


1.1  Project  Overview 

One  of  the  most  challenging  activities  for  a  distance  education  facilitator  is  to  be 
responsive  to  student  needs  and  customize  the  learning  experience  to  the  student’s 
individual  learning  style,  help  socialize  the  student  to  the  institution,  and  keep  the 
learner  engaged  in  spite  of  the  isolated  environment.  To  achieve  these  goals,  Defence 
Research  and  Development  Canada  (DRDC)  -  Toronto  has  initiated  an  Applied 
Research  Project  (ARP)  to  investigate  mechanisms  to  improve  efficiency  and 
effectiveness  for  the  Canadian  Forces  (CF)  distance  learning  capability.  Two 
mechanisms  to  facilitate  the  learning  experience  are  adaptive  learning  and  intelligent 
tutoring  technologies. 

Adaptive  learning  in  the  context  of  e-learning  involves  creating  a  learning  experience 
that  purposely  adjusts  to  various  conditions  by  adapting  to  individual  students’  learning 
needs  based  on  measures  such  as  their  individual  behaviour,  workload,  or 
performance.  Similarly,  ITSs  are  self-regulating  systems  for  the  control,  delivery,  and 
assessment  of  learning  contents.  Complex  algorithms  are  designed  to  rely  on  feedback 
from  the  learner’s  performance,  prior  exposure  to  knowledge,  and  learning  rate  to 
deliver,  evaluate,  and  react  according  to  pedagogical  principles,  educational  goals,  and 
implementation  tools. 

To  make  the  technologies  effective  in  a  distance  learning  environment,  the  learning 
platform  and  adaptation  mechanisms  to  customize  the  learning  experience  need  to  be 
investigated.  The  CF  can  also  benefit  from  mobile  devices  in  theatre  for  continuation 
and  re-fresher  training.  The  execution  of  this  work  has  been  contracted  to  CAE  PS  who 
will  survey  and  synthesise  the  body  of  knowledge  on  principles  and  best  practices  for 
adaptation  mechanisms  and  mobile  learning  technologies  in  education  and  training.  The 
results  of  this  study  will  be  used  to  develop  a  technology  implementation  and  validation 
plan. 

In  order  to  support  these  goals,  the  following  tasks  were  conducted  during  the  first 
phase  of  the  ITS  project  (January  to  March  2010;  see  Figure  1-1): 
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1 .  Literature  Review  on  Suitable  Adaptation  Mechanism  and  Mobile  Learning 
Technologies  for  Adaptive  Learning  and  Intelligent  Tutoring  Environments.  To 
review  and  identify  suitable  adaptation  mechanism  and  mobile  learning 
technologies  for  adaptive  learning  and  intelligent  tutoring  methods  that  would 
improve  CF  distance  learning  capabilities. 

2.  Requirement  and  Stakeholder  Analysis  for  the  CF  Counter-IED  Training 
Courses.  To  investigate  the  requirements  of  stakeholders  of  the  CF  C-IED 
Disposal  Operator  Training  course  for  the  development  of  adaptive  learning 
technology  integration  and  validation  plans. 


Figure  1-1:  The  relationship  between  tasks  (and  previous  research)  conducted  within  the 

ITS  project.  Current  tasks  are  indicated  in  bold. 

This  report  describes  the  results  of  the  second  task;  the  Requirement  and  Stakeholder 
Analysis  for  the  CF  Counter-IED  Training  Courses.  Parallel  research  efforts  have 
explored  available  technology  and  methodology  for  implementing  real-time  adaptive 
mechanisms  into  a  CF  training  environment.  This  report  is  founded  on  several 
previously  completed  CAE  PS  reports  concerning  learning  styles  (Provonost,  Roberts 
and  Banbury,  2008),  intelligent  adaptive  systems  (Banbury,  Gauthier  and  Scipione, 
2007),  and  suitable  adaptive  learning  and  intelligent  tutoring  technologies  (Banbury, 
Roberts,  Hartlen  and  Unrau,  2009). 

These  initial  reports  provided  a  high-level  overview  of  the  requirements  and  availability 
of  intelligent  adaptive  interfaces  and  techniques,  such  as  determining  the  aspects  of 
cognition  that  have  been  used  to  measure  mental  workload.  Each  subsequent  report 
sought  to  narrow  the  overall  scope  in  each  respective  area  in  order  to  focus  on  a 
specific  type  and  context  for  which  the  intelligent  interface  would  be  implemented. 
Specifically,  after  learning  of  the  overall  types  of  research  being  conducted  on  adaptive 
learning,  the  second  and  third  reports  investigated  specific  technologies  that  could 
possibly  adapt  to  the  human  learner,  as  well  as  which  DND  learning  environment  would 
be  the  ideal  platform  for  proof  of  concept.  These  studies  are  described  briefly  in  the 
following  section. 
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1.2  Previous  Research  Conducted  by  CAE  PS 

Results  of  CAE  PS’s  previous  research  produced  a  number  of  recommendations  for  the 
implementation  of  adaptive  learning  and  intelligent  technologies  into  the  IEDD  Operator 
Course.  The  identification  of  recommendations  for  implementing  adaptive  learning  and 
intelligent  tutoring  technologies  is  also  the  basis  of  this  report;  however,  efforts  have 
been  taken  to  tailor  the  recommendations  to  meet  the  specific  needs  of  the  IEDD 
Operator  Course.  To  alleviate  the  need  to  refer  to  numerous  documents,  the  following 
sections  recap  the  main  knowledge  gained  from  the  previous  CAE  contracts,  as  they 
are  the  underlying  foundation  supporting  the  stakeholder  analysis. 


1.2.1  Intelligent  Adaptive  Interfaces 

The  goal  of  this  literature  research  was  to  provide  support  to  establish  guidelines  for  the 
effective  design  of  Intelligent  Adaptive  Systems  (IASs)  through  the  review  of 
frameworks,  analysis  tools  and  processes  for  IASs,  and  through  the  provision  of  general 
design  recommendations  and  guidelines  for  the  development  of  IASs,  with  particular 
emphasis  on  the  operator  machine  interface  (OMI).  Combining  design  methodologies 
from  both  Human  Computer  Interaction  (HCI)  and  Human  Factors  (HF)  fields, 
conceptual  and  design  frameworks  were  also  developed  to  provide  guidelines  for  the 
design  and  implementation  of  IASs.  Finally,  a  number  of  criteria  that  can  be  used  to 
select  appropriate  analytical  techniques  and  design  approaches  were  also  developed. 

After  reviewing  the  approaches  concerned  with  the  design  of  an  intelligent  adaptive 
system,  a  generic  conceptual  architecture  was  developed.  It  has  the  following  four 
components,  which  are  common  to  all  developed  and  developing  IASs: 

•  Situation  Assessment  and  Support  System.  This  provides  information  about  the 
objective  state  of  the  aircraft/vehicle/system  within  the  context  of  a  specific 
mission,  and  uses  a  knowledge-based  system  to  provide  assistance  (e.g., 
automate  tasks)  and  support  to  the  operator. 

•  Operator  State  Assessment.  This  provides  information  about  the  objective  and 
subjective  state  of  the  operator  within  the  context  of  a  specific  mission  relating  to 
real-time  analysis  of  the  psychological,  physiological  and/or  behavioural  state  of 
the  operator  (e.g.,  continuous  monitoring  of  workload,  inferences  about  current 
attentional  focus,  ongoing  cognition,  visual  and  verbal  processing  load),  and 
intentions  using  extensive  a  priori  operator  knowledge  (e.g.,  models  of  human 
cognition,  control  abilities,  and  communication). 

•  Adaptation  Engine.  This  utilizes  the  higher-order  outputs  from  Operator  State 
Assessment  and  Situation  Assessment  systems,  as  well  as  other  relevant 
aircraft/vehicle/system  data  sources,  to  maximize  the  match  between 
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aircraft/vehicle/system  state,  operator  state,  and  the  tactical  assessments 
provided  by  the  Situation  Assessment  system. 

•  Operator  Machine  Interface  (OMI).  The  means  by  which  the  operator  interacts 
with  the  aircraft/vehicle/system  in  order  to  satisfy  mission  tasks  and  goals.  This  is 
also  the  means  by  which,  if  applicable,  the  operator  interacts  with  the  intelligent 
adaptive  system  (e.g.,  a  tasking  interface  manager). 

Figure  1-2  illustrates  how  all  four  components  operate  within  the  context  of  a  closed- 
loop  system:  a  feedback  loop  re-samples  operator  state  and  situation  assessment 
following  the  adaptation  of  the  OMI  and/or  automation.  The  goal  is  to  adjust  the  level 
of  adaptation  so  that  optimal  operator  states  (e.g.,  performance,  workload  etc)  are 
attained  and  maintained. 


■  Knowledge  of  mission  plans/goals 

■  Knowledge  of  mission  time-lines 

■  Knowledge  of  mission  tasks/activities 

■  Knowledge  of  mission  environment 
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■  Model  of  human  interaction 


Figure  1-2:  IAI  Conceptual  Architecture  for  Intelligent  Adaptive  Systems 


The  literature  research  pertaining  to  IASs  demonstrated  that  when  automation  and 
interface  adaptation  that  are  implemented  dynamically  and  intelligently  (i.e.,  in  response 
to  changing  task  demands  placed  upon  the  operator),  the  chief  benefits  of  automation 
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(e.g.,  workload  regulation)  can  be  realised  without  most  of  the  drawbacks  associated 
with  conventional  or  static  automation  (e.g.,  loss  of  Situation  Awareness).  The  review 
achieved  of  the  following  goals: 

•  Identified  the  advantages,  disadvantages  and  applicability  of  development 
frameworks,  analysis  methodologies,  design  approaches,  and  operator-state 
monitoring  approaches; 

•  Made  some  progress  in  unifying  the  hitherto  independent  HF  and  HCI 
approaches  to  the  development  of  IASs  by  providing  a  generic  conceptual 
framework  and  a  generic  conceptual  architecture  which  map  to  both  approaches 
by  focusing  on  system  functionality  and  capability;  and, 

•  Integrated  design  methodologies  from  both  HCI  and  HF  fields  and  develop 
guidance  for  developers  to  assist  in  the  successful  design,  development  and 
implementation  of  IASs,  with  particular  emphasis  on  the  development  of  the  OMI. 
A  number  of  criteria  for  the  selection  of  appropriate  analytical  and  design 
methodologies  were  also  recommended.  The  present  report  utilises  the  decision 
trees  developed  by  Banbury  et  al.,  (2007)  to  select  the  most  appropriate  tools 
and  methodologies  for  the  IEDD  Operator  Course  ITS. 

1.2.2  Cognitive  Learning  Styles 

Students  approach  learning  tasks  and  interact  with  learning  environments  in  different 
ways;  they  develop  a  specific  set  of  learning  behaviours,  described  as  ‘learning  styles’ 
that  they  are  comfortable  with  (Entwistle,  1981 ).  The  objective  of  this  review  was  to 
survey  and  synthesize  the  body  of  knowledge  on  student  learning  styles  and  develop 
recommendations  on  how  to  integrate  knowledge  of  the  learning  styles  of  the  learner 
into  adaptive  learning  and  intelligent  tutoring  technologies  to  facilitate  a  positive  learning 
experience  to  improve  CF  distance  learning  capabilities. 

The  report  reviews  theories  of  student  learning  styles,  relevant  technologies  to  best 
exploit  the  different  styles  in  education  and  training,  principles,  and  methodologies  of 
implementations  in  learning  environments,  particularly  in  distance/e-learning 
environments.  The  flexibility  offered  by  such  environments  should  enhance  learning; 
allowing  students  to  develop  personal  navigation  patterns  and  interaction  behaviour  that 
reflects  their  own  cognitive  characteristics. 

The  review  concluded  that  models  based  on  learning  styles  have  the  potential  to 
provide  the  theoretical  background  for  designing  educational  systems,  build  their  user 
model  and  functionality,  and  guide  decisions  about  what  the  system  should  offer  to 
students  with  different  styles  in  the  case  of  adaptive  educational  systems.  The  final 
section  of  the  report  provided  recommendations  based  on  the  literature  review,  as  well 
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as  integrated  and  discussed  some  case  studies  that  have  had  positive  experiences  after 
integrating  knowledge  of  the  student’s  learning  styles  into  Intelligent  Tutoring  Systems 
(ITSs). 

The  literature  research  concluded  the  following: 

•  While  the  overall  assessment  of  learning  style  models  has  been  rather 
inconclusive  it  should  not  undermine  their  potential  to  support  e-learning 
technologies.  Indeed,  any  increment  in  user  satisfaction  or  perceived  usability  of 
ITSs  may  justify  the  means  undertaken. 

•  Another  important  aspect  to  consider  is  the  ratio  of  benefits  relative  to  costs 
incurred  for  the  prototyping,  development,  implementation,  maintenance,  and 
collateral  investments  related  to  a  system  such  as  training  and  updates.  The 
implementation  of  intelligent  tutoring  systems  that  adapt  to  the  learning  style  of 
the  student  is  far  from  simple,  and  the  benefits  of  its  realization  may  not  outweigh 
its  costs;  particularly  in  light  of  the  rather  minimal  benefits  observed  our  review  of 
the  learning  styles  literature. 

•  Proposal  of  three  design  options  -  (i)  an  overarching  system  to  be  built  bottom- 
up,  (ii)  staying  with  current  technologies,  or  (iii)  amending  existing  technologies  to 
accommodate  new  interests  -  need  to  be  carefully  assessed  based  on  a  balance 
of  resources  and  requirements. 

1.2.3  Feasibility  of  Adaptive  Learning  and  Intelligent  Tutoring  Technologies 

The  feasibility  study  of  adaptive  learning  and  intelligent  tutoring  technologies  had  two 
main  objectives:  The  first  was  to  review  and  identify  suitable  Adaptive  Learning  and 
Intelligent  Tutoring  technologies  that  could  be  integrated  into  the  DND  Learn 
environment  to  improve  CF  distance  learning  capabilities.  The  second  objective  was 
identify  a  suitable  CF  training  course  that  can  demonstrate  the  benefits  of  these 
technologies,  together  with  identifying  suitable  integration  components  and  designing  an 
implementation  and  evaluation  framework. 

Objective  1 :  The  results  of  the  literature  review  demonstrated  a  range  of  maturing 
technologies  that  can  be  usefully  applied  to  CF  e-learning  initiatives.  The  components 
that  could  be  incorporated  are: 

•  Eye  Tracking.  Eye  tracking  affords  insight  into  the  student’s  locus  of  attention. 
This  information  can  be  used  by  both  the  instructor  and  the  system.  The  visual 
significance  attributed  by  the  student  to  features  in  a  simulated  scene  can  be 
compared  to  significances  obtained  from  the  training  objectives  or  from  tracking 
data  recorded  from  subject  matter  experts  (SMEs). 
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•  Electroencephalography  (EEG).  Data  can  be  correlated  with  training  content 
events  to  develop  workload  measures.  The  workload  component  of  the  student 
model  can  then  be  updated  using  this  information. 

•  Heart  Rate  Variability  (HRV).  This  measure  can  be  used  to  infer  overall  levels  of 
stress,  and  students’  affective  level  of  arousal  in  reaction  to  presented  stimuli. 

•  Galvanic  skin  response  (GSR).  Also  known  as  electrodermal  response  (EDR), 
psychogalvanic  reflex  (PGR),  or  skin  conductance  response  (SCR),  this 
technology  can  be  used  as  an  indicator  of  the  students  level  of  excitement  or 
relaxation. 

•  Student  Interaction  Measures.  Information  from  the  training  delivery  can  be 
compared  to  knowledge  of  the  training  content  to  obtain  measures  of  student 
proficiency,  pace  and  behaviour.  Where  possible,  this  information  should  be 
handled  in  a  fashion  that  does  not  preclude  Sharable  Content  Object  Reference 
Model  (SCORM)  compliance.  However,  the  implementation  limitations  of 
SCORM  around  virtual  simulation  dictate  that  full  SCORM  compliance  will  not  be 
possible. 

Objective  2:  The  results  of  the  review  demonstrated  that  the  IEDD  Operator  Course  was 
the  most  suitable  CF  training  course  for  the  integration  of  adaptive  learning,  intelligent 
tutoring  and  cognition  monitoring  technologies.  A  number  of  factors  were  taken  into 
account  by  the  analysis,  for  example: 

•  The  extent  to  which  the  course  is  appropriate  for  the  integration  of  advanced 
learning  technologies.  For  example,  it  would  be  beneficial  to  the  study  if  the 
course  delivery  can  be  usefully  implemented  within  a  variety  of  environments  to 
allow  the  greatest  flexibility  for  adapting  both  course  content  and  delivery 
method; 

•  The  extent  to  which  the  course  includes  the  training  of  complex  skills.  For 
example,  it  would  be  beneficial  to  the  study  if  the  course  content  included  the 
instruction  of  complex  cognitive  skills  (procedural  knowledge)  -  as  opposed  to 
‘how  to’  (declarative)  knowledge  -  to  allow  the  greatest  flexibility  for  adapting 
both  course  content  and  delivery  method  based  on  the  student’s  learning  style; 

•  The  extent  to  which  the  course  has  high  student  failure  rate.  For  example,  it 
would  be  beneficial  to  the  study  if  there  is  a  relatively  high  student  failure  rate  in 
the  course  as  this  would  mitigate  any  ceiling  effects  that  might  be  observed  when 
evaluating  the  utility  (i.e.,  ‘added  value’)  of  adaptive  learning  technologies. 
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The  IEDD  Operator  Course  is  designed  to  instruct  soldiers  in  skills  and  procedures 
relating  to  the  identification  and  disposal  of  lEDs  in  operational  and  domestic  settings 
together  with  related  administrative  duties  in  support  of  IEDD  operations. 

1.3  Scope  and  Objectives 

The  objective  of  this  task  is  to  investigate  the  requirements  of  key  stakeholders  of  the 
CF  IEDD  Operator  Course  for  the  implementation  and  evaluation  of  adaptive  learning 
and  intelligent  tutoring  technologies  within  that  course.  Given  the  high  failure  rate  in  the 
existing  IEDD  Operator  Course,  problematic  aspects  of  the  course  was  one  of  the  foci 
of  the  Stakeholder  Analysis.  The  discussion  with  stakeholders  focused,  in  part,  on  the 
feasibility  of  demonstrating  the  capability  of  intelligent  tutoring  technology  within  the 
IEDD  Operator  Course. 

1.4  This  Document 

The  structure  of  this  document  is  described  below: 

•  Section  2.  Presents  the  results  of  the  Stakeholder  Analysis  interviews  conducted 
between  the  22nd  January  2010  and  the  23rd  February  2010.  This  section  also 
describes  the  recommendations  for  implementing  adaptive  learning  and 
intelligent  tutoring  technologies  into  a  suitable  CF  distance-learning  course; 

•  Section  3.  Describes  the  implementation  plan  for  the  IEDD  Operator  Course  ITS, 
in  terms  of  its  content  and  capabilities.  This  section  also  describes  the  notional 
architecture  of  the  IEDD  ITS  in  more  detail,  as  well  as  the  schedule  for  the 
design,  implementation  and  evaluation  of  the  ITS; 

•  Section  4.  Describes  the  evaluation  plan  for  the  IEDD  Operator  Course  ITS  in 
terms  of  the  methodology  and  approximate  schedule  for  activities  pertaining  to 
the  evaluation  of  the  ITS;  and, 

•  Section  5.  Summarises  the  results  of  the  stakeholder  analysis,  and  provides 
recommendations  for  the  next  steps  for  both  the  implementation  plan  and 
evaluation  plan. 
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2  STAKEHOLDER  ANALYSIS 

This  section  presents  the  results  of  the  Stakeholder  Analysis  interviews  conducted 
between  the  22nd  January  2010  and  the  23rd  February  2010.  Following  a  general 
introduction,  this  section  describes  the  methodology  utilised  for  the  analysis,  followed  by 
the  results.  The  final  part  of  this  section  describes  the  recommendations  for 
implementing  adaptive  learning  and  intelligent  tutoring  technologies  into  a  suitable  CF 
distance-learning  course. 

2.1  Introduction 

The  objective  of  the  Stakeholder  Analysis  was  to  investigate  the  requirements  of 
stakeholders  of  the  CF  C-IED  Disposal  Operator  Training  course  for  the  development  of 
adaptive  learning  technology  integration  and  validation  plans.  The  Stakeholder  Analysis 
comprised  two  components: 

1 .  Investigation  of  Current  Training  Tools  for  C-IED  Task  Forces.  This  component  of 
the  analysis  sought  to  identify  and  scope:  (i)  stakeholders  and  their  requirements; 
(ii)  their  existing  instructional  content,  and  (iii)  their  plans  for  future  content  and 
training  directions  and  requirements.  The  following  stakeholders  were  identified 
by  the  Scientific  Authority: 

a.  Counter-IED  Task  Force  /  Training,  National  Defence  Headquarters 
(NDHQ)  (Ottawa,  ON); 

b.  IEDD  Operator  Training  and  C-IED  Defeat  the  Device  Center  of 
Excellence,  CF  School  of  Military  Engineering  (CFB  Gagetown,  NB) 

c.  C-IED  Attack  the  Network,  Combat  Training  Centre  (CFB  Gagetown,  NB); 
and, 

d.  Counter-IED  Technology  Demonstration  Project  (TDP),  DRDC  Toronto 
(Toronto,  ON). 

2.  Investigation  of  E-Learning  Environments  and  Learning  and/or  Content 
Management  Systems.  This  component  of  the  analysis  sought  to  identify  the 
suitability  of  e-learning  environments  and  learning  material  authoring  software 
such  as  Learning  and/or  Content  Management  Systems,  and  assess  the 
integration  feasibility  of  different  application  software  in  multiple  learning 
environments.  The  following  stakeholders  were  identified  by  the  Scientific 
Authority: 


28  May  2010  -9-  5045-1  Version  02 

©  Her  Majesty  the  Queen  as  represented  by  the  Minister  of  National  Defence,  2010 
©  Sa  majeste  la  reine,  representee  par  le  ministre  de  la  Defense  nationale,  2010 


CAE 


Stakeholder  Analysis  for  C-IED  T raining  Courses 


a.  Institute  for  Information  Technology,  National  Research  Council  of  Canada 
(Fredericton,  NB); 

b.  Directorate  of  Land  Synthetic  Environments  (CFB  Kingston,  ON);  and, 

c.  Directorate  of  Learning  Innovation,  Canadian  Defence  Academy  (CFB 
Kingston,  ON). 

The  details  regarding  the  timing  and  points  of  contact  for  each  of  the  Stakeholder 
Analysis  meetings  are  presented  in  Appendix  A.  The  following  sections  describe  the 
work  conducted  under  the  auspices  of  the  stakeholder  analysis. 

2.2  Stakeholder  Meetings 

Prior  to  the  commencement  of  the  stakeholder  meetings,  CAE  PS  identified  a  number  of 
discussion  points  in  the  context  of  CF  distance  learning,  and  circulated  them  in  advance 
of  the  meetings.  The  discussion  points  encompassed  the  two  components  of  the 
stakeholder  analysis  identified  in  the  previous  section.  Specifically: 

1 .  Training  courses  supported.  To  gain  an  appreciation  of  what  Counter-1  ED 
courses  each  stakeholder  is  supporting,  the  discussions  were  focused  on  those 
courses  that  require  simulation-based  training,  and/or  training  of  complex 
cognitive  skills  (e.g.,  Situation  Awareness  and  decision  making).  Given  the 
recommendations  of  the  Banbury  et  al. ,  (2009)  report,  particular  emphasis  was 
placed  on  the  IEDD  Operator  Course; 

2.  Training  content  developed.  For  those  courses  identified  above,  the  discussions 
were  focused  on  what  types  of  content  has  already  been  created,  and  in  what 
format.  The  exploitation  of  existing  content  for  the  development  of  the  ITS  course 
was  also  discussed; 

3.  Content  development  tools  used.  The  discussions  were  focused  on  the  content 
development  tools  that  have  been  used  in  the  past,  those  used  currently,  and 
those  that  might  be  used  in  the  future.  In  addition,  any  capability  gaps  between 
what  content  the  stakeholder  would  like  to  develop  and  what  the  tools  of  the 
stakeholder  could  currently  support  was  also  discussed.  This  knowledge  will  the 
ITS  project  team  determine  which  development  tools  should  be  adopted. 

4.  Future  training  delivery  plans.  The  discussions  were  focused  on  the  future  role  of 
serious  game-based  technologies,  and  mobile  computing  (e.g.,  iTouch,  palm 
PCs,  hand-held  gaming  devices)  in  future  plans  for  delivery  of  training  content; 
and, 
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5.  Past,  present  and  future  challenges.  Finally,  the  discussions  focused  on  any 
insights  into  the  kinds  of  challenges  that  have  experienced  by  the  stakeholder  in 
the  past,  those  experienced  currently,  and  those  that  they  expect  to  experience 
in  the  future.  As  well  as  informing  the  decision  making  about  how  the  ITS  course 
might  be  implemented,  the  ITS  project  team  is  keen  to  leverage  any  lessons- 
learnt  during  the  ITS  project,  to  help  the  stakeholder  overcome  their  challenges. 

This  list  of  discussion  points  was  not  exhaustive;  in  fact,  the  discussions  covered  many 
related  topics.  The  content  of  the  discussions  were  recorded  in  Mindjet’s 
MindManager™  application. 


2.2.1  Results 

The  following  section  provides  a  top-level  summary  of  the  discussions  had  during  each 
stakeholder  meeting.  Detailed  notes  pertaining  to  each  of  the  stakeholder  meetings  are 
presented  in  the  accompanying  MindMap. 


2.2. 1.1  Counter-IED  Task  Force  /  Training  (NDHQ,  Ottawa) 

The  following  issues  were  discussed  with  Maj  Schurman  and  MWO  Crosby  during  this 
stakeholder  meeting: 

•  IEDD  Operator  Course.  Participants  discussed  several  aspects  of  the  IEDD 
Operator  Course,  including  general  course  information,  and  potential  avenues  in 
which  adaptive  learning  and  intelligent  tutoring  technologies  might  be 
implemented.  Specifically: 

o  General  course  enrolment  information.  Students  are  required  to  have  at 
least  four  years  of  service  before  taking  the  course.  The  Advanced 
Munitions  Disposal  course  is  a  pre-requisite  for  some  occupations.  Navy 
divers  are  the  most  successful  trade,  although  the  reasons  as  to  why  this 
is  the  case  are  not  clear. 

•  IEDD  Operator  mission  scenario.  Participants  discussed  several  aspects  of 
typical  IEDD  mission  scenarios.  Specifically: 

o  IEDD  team  information.  The  IEDD  team  comprises  two  operators;  the 
Number  1  that  conducts  the  disposal  tasks,  and  the  Number  2  that  assists 
the  Number  2.  The  team’s  Commanding  Officer  (CO)  will  give  the  initial 
briefing  to  the  team,  and  will  also  answer  any  questions  (e.g.,  relevant 
intelligence)  that  the  IEDD  team  might  have. 
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o  IEDD  Operator  Course  scenarios.  The  course  uses  a  number  of  scenarios 
to  train  and  evaluate  students.  At  present,  nine  scenarios  are  run;  two  are 
based  on  domestic  operations,  and  seven  are  based  on  Afghanistan 
operations. 

o  IEDD  mission  scenario.  A  typical  IEDD  mission  scenario  was  discussed  in 
detail.  It  was  suggested  that  this  type  of  scenario  should  be  used  for  the 
ITS  project.  Further  analysis  is  required  to  decompose  the  mission 
scenario  into  sufficient  detail  in  order  to  identify  the  cognitive  components 
(e.g.,  decision  making  requirements,  Situation  Awareness  ‘elements’)  and 
learning  points  (i.e.,  Training  Needs  Analysis).  Atypical  mission  comprises 
the  following  high-level  functions: 

■  Receive  briefing  -  Communication  with  IEDD  cell  after  blast  or 
suspected  IED.  Receive  briefing  before  departure.  Additional 
briefings  en  route. 

■  Arrive  on  scene  and  conduct  threat  assessment  -  Discuss  situation 
with  On  Scene  Commander  (OSC)  and  question  witnesses  to 
assess  threat  and  determine  appropriate  RSP  (Render  Safe 
Procedure). 

■  Conduct  disposal  -  Procedures  followed  according  to  which  RSP 
was  determined.  This  plan  might  be  modified  in  light  of  new 
information  acquired  during  the  disposal  process  (e.g.,  finding 
evidence  of  other  IED  components). 

■  Conduct  After  Action  Review  (AAR)  -  Receive  feedback  on  the 
IEDD  team’s  decision  making  and  workload  management. 

•  Links  with  other  CF projects  / initiatives.  NDHQ  is  the  strategic  headquarters  of 
counter-IED  efforts.  Participants  discussed  how  the  Counter-IED  Task  Force 
project  is  linked  with  other  CF  projects  and  initiatives.  These  links  are  as  follows: 

o  CFB  Gaqetown.  New  training  cell  and  instructors  at  the  Combat  Training 
Centre  (Maj  Durant)  and  the  School  of  Military  Engineering  (Capt  Gilbert). 

o  Directorate  of  Land  Requirements  (DLR).  Interaction  as  part  of  the 
Counter-IED  Task  Force.  NDHQ  is  acting  as  point  of  contact  for  these 
efforts. 

o  Directorate  of  Land  Synthetic  Environments  (DLSE).  Minimal  contact  with 
Capt  Taff  at  the  DLSE,  although  this  is  likely  to  change  soon  in  order  to 
roll-out  IED  efforts  to  the  rest  of  the  Army.  It  is  important  that  the 
technologies  adopted  are  compatible  with  their  plans. 
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2.2. 1.2  IEDD  Operator  Training  and  C-IED  Defeat  the  Device  Center  of 
Excellence  (CF  School  of  Military  Engineering,  CFB  Gagetown) 

The  following  issues  were  discussed  with  Capt  Gilbert,  WO  Drolet  and  WO  Lefort  (IEDD 
Operator  Training  -  Instructor)  during  this  stakeholder  meeting: 

•  Operational  conditions.  IEDD  operators  in  Afghanistan  are  currently  responding 
to  3  to  4  call-outs  per  day.  As  a  result,  there  is  an  enormous  amount  of  pressure 
on  the  IEDD  operators,  and  they  frequently  experience  burn-out. 

•  IEDD  Operator  Course.  Participants  discussed  several  aspects  of  the  IEDD 
Operator  Course,  including  general  course  information,  and  potential  avenues  in 
which  adaptive  learning  and  intelligent  tutoring  technologies  might  be 
implemented.  This  information  is  presented  in  Section  2.4.1  IEDD  Operator 
Course  Analysis. 

•  IEDD  Operator  ITS  mission  scenario.  Participants  discussed  the  potential  of 
adding  an  additional  practice  scenario  before  the  evaluations.  This  scenario 
would  provide  the  ITS  project  team  the  opportunity  to  evaluate  the  impact  of 
adaptive  learning  and  intelligent  tutoring  technologies  within  the  IEDD  Operator 
Course,  that  would  both,  (i)  afford  the  ITS  project  team  with  an  opportunity  to 
fully-exercise  a  range  of  adaptive  learning  and  intelligent  tutoring  technologies, 
and  (ii)  provide  the  greatest  possible  benefit  to  students  on  the  IEDD  Operator 
Course.  This  information  is  presented  in  Section  2.4.1  IEDD  Operator  Course 
Analysis. 

•  Evaluation  of  ITS  technologies  with  the  IEDD  course.  Participants  discussed  the 
constraints  pertaining  to  conducting  the  evaluation  of  the  ITS  scenario  with 
students  from  the  course.  No  control  condition  in  which  some  students  have  a 
'lesser  learning  experience'  will  be  acceptable.  Manipulation  of  mechanisms  of 
adaptation  should  not  change  the  content  just  how  it  is  delivered,  with  no  impact 
on  both  the  quality  and  the  duration  of  the  training. 

•  Links  with  other  CF  projects  /  initiatives.  Participants  discussed  how  the  Defeat 
the  Device  Centre  of  Excellence  and  the  IEDD  Operator  Course  are  linked  with 
other  CF  projects  and  initiatives.  These  links  are  as  follows: 

o  Defeat  the  Device  Centre  of  Excellence  (CFB  Gagetown).  IEDD  operators 
will  often  try  to  render-safe  the  device  through  deactivation  rather  than 
destruction.  In  doing  so,  forensic  analysis  can  be  undertaken  to  inform  and 
refine  disposal  procedures,  as  well  as  identify  the  network  through  which 
the  device  was  funded,  constructed,  transported  and  installed.  The  Defeat 
the  Device  Centre  of  Excellence  has  been  using  Virtual  Battle  Space  2 
(VBS2)  to  create  training  videos  for  IED  awareness;  albeit  not  yet  for  the 


28  May  2010  -  13-  5045-1  Version  02 

©  Her  Majesty  the  Queen  as  represented  by  the  Minister  of  National  Defence,  2010 
©  Sa  majeste  la  reine,  representee  par  le  ministre  de  la  Defense  nationale,  2010 


Stakeholder  Analysis  for  C-IED  T raining  Courses 


CAE 


IEDD  Operator  Course.  These  videos  were  developed  by  the  Combat 
Training  Centre,  CFB  Gagetown. 

o  Combat  Training  Centre  (CFB  Gagetown).  Collaboration  with  Capt  Fox, 
who  is  using  VBS2  to  create  training  videos  for  a  range  of  mounted  and 
dis-mounted  tactics;  including  IED  awareness. 

o  C-IED  TDP  (DRDC  Toronto).  Collaboration  with  Dr  Jerzy  Jarmasz  and 
Sgt.  Wojtarowicz  regarding  development  of  VBS2  environments  relating  to 
C-IED. 


2.2. 1.3  C-IED  Attack  the  Network  (Combat  Training  Centre,  CFB  Gagetown) 

The  following  issues  were  discussed  with  Maj  Gimby  and  Capt  Fox  during  this 
stakeholder  meeting: 

•  Simulation-based  tools  used.  Participants  discussed  what  simulation-based  tools 
were  currently  being  used  by  the  Combat  Training  Centre. 

o  VBS2.  Current  development  using  VBS2  is  restricted  by  classification; 
content  is  now  being  developed  by  intern  students  in  partitions  to  avoid 
classification  issues.  VBS2  is  being  used  to  create  training  videos.  For 
example,  the  re-construction  of  scenarios  that  happened  in  the  last  30 
days  to  enable  knowledge  sharing.  This  is  a  cost-effective  means  of 
producing  training  videos  and  content. 

o  Other  tools.  The  Combat  Training  Centre  is  exploring  ‘parallel’  paths  to 
develop  content  in  other  3D  modeling  tools  due  to  issues  relating  to  VBS 
Gold  version  licensing. 

o  Flash.  Content  is  being  developed  using  Flash  within  DND  Learn. 

•  Simulation-based  content  generated.  Participants  discussed  what  simulation- 
based  content  has  been  generated  that  might  be  re-purposed  for  the  ITS  project. 

o  VBS2  models  in  development.  Seven  training  videos  have  been  produced 
and  have  been  given  to  CAE  PS.  The  videos  are  set  in  Afghanistan-like 
situations;  with  models  pertaining  to  CF  vehicles,  personnel,  and  terrain 
features. 

•  Future  modeling  efforts.  The  team  would  like  to  integrate  real  terrain  and  building 
imagery  into  the  simulation  software. 
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•  Training  courses  supported.  The  participants  discussed  the  Tactical  Exploitation 
course  which  is  run  under  the  auspices  of  the  Defeat  the  Network  initiative.  This 
is  a  three  week  course,  which  includes  eight  days  of  scenarios  and  evaluation. 
The  course  focuses  on  forensics,  evidence  and  tactical  questioning  (e.g.,  pattern 
matching,  and  social  networking).  Although  this  course  has  a  100  percent  pass 
rate  at  present,  the  participants  noted  that  a  training  scenario  in  a  simulated 
environment  would  be  useful  to  keeps  these  skills  refreshed. 

•  Links  with  other  CF  projects  / initiatives.  Participants  discussed  how  the  Defeat 
the  Device  Centre  of  Excellence  and  the  IEDD  Operator  Course  are  linked  with 
other  CF  projects  and  initiatives.  These  links  are  as  follows: 

o  C-IED  TDP  (DRDC  Toronto).  Collaboration  with  Dr  Jerzy  Jarmasz  and 
Sgt.  Wojtarowicz  regarding  development  of  VBS2  environments  relating  to 
C-IED.  For  example,  the  Environmental  Familiarisation  and  Indicator 
Trainer  (EFIT). 

o  School  of  Military  Intelligence  (CFB  Kingston).  Point  of  contact  Maj  Terfry 
regarding  the  Tactical  Questioning  course. 

o  CF  School  of  Communications  and  Electronics.  Point  of  contact  Capt 
Shea  regarding  Counter  IED  initiatives  (Defeat  the  Device). 

o  Army  Learning  Support  Centre  (ALSO.  Point  of  contact  Maj  Batty  (TDOs 
Maj  Standish  and  Lt.  Navy  Paula  Jardine).  ALSC  are  leaders  in  the 
development  of  VBS  content  and  learning  strategies.  VBS  products  are 
also  developed  by  student  interns. 

o  Defeat  the  Device  Centre  of  Excellence  (CFB  Gaqetown).  From  an 
Electronic  Warfare  (EW)  perspective. 


2.2. 1.4  Institute  of  Information  Technology  (NRC,  Fredericton) 

The  following  issues  were  discussed  with  Dr  Rod  Savoie,  Dr  Marc-Allain  Mallet  and  Dr 
Bruce  Spencer  during  this  stakeholder  meeting: 

•  Related  Projects.  Participants  discussed  several  projects  at  the  NRC  that  are 
relevant  to  adaptive  learning  and  intelligent  tutoring  technologies.  These  projects 
are  as  follows: 

o  Personal  Learning  Environment  (P-Learn)  -  learner-oriented  access  to 
Learning  Content  Management  Systems  (LCMS),  such  as  Desire2Learn; 

o  Cognitive  modeling; 
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o  Natural  language  processing; 

o  Data  and  text  mining  summary  tool; 

o  WEBDAV  -  an  automated  web-authoring  tool  for  content  development, 
integrated  into  DND  Learn;  and, 

o  Doctor-patient  interview  adaptive  learning  system. 

2.2. 1.5  Directorate  of  Land  Synthetic  Environments  (DLSE)  (CFB  Kingston) 

The  following  issues  were  discussed  with  Capt  Taff  and  Dr  Roman  during  this 
stakeholder  meeting: 

•  DLSE  Background.  The  DLSE  is  responsible  for  the  collective  training  of  the 
army  for  section  level  and  above.  Capt  Taff  is  the  project  manager  for  the 
simulators  at  DLSE,  which  includes  responsibility  for  the  purchasing  of  VBS2  and 
the  software  development  team  (approximately  25  persons  in  Kingston  and 
Ottawa).  The  development  team  creates  interfaces  and  smaller  tools,  and 
provides  support  to  DRDC  projects  and  development;  for  example,  VBS2  support 
and  the  provision  of  standardised  development  approaches.  Capt  Taff  provides  a 
good  point  of  contact  for  all  simulation  development  initiatives  (including  VBS2) 
within  the  Army. 

•  Simulation-based  tools  used  or  examined.  DLSE  has  looked  into  Olive’s  Forterra 
and  Linden  Lab’s  Second  Life  virtual  environments.  However,  both  approaches 
were  not  followed  given  the  requirement  to  acquire  open-source  and  simulation- 
independent  architectures  in  order  to  be  future-proof  as  much  as  possible. 

o  VBS2.  VBS2  has  been  used  for  counter-IED  training;  for  example  convoys 
and  recognising  IED  indicators.  Nearly  all  this  training  content  is 
unclassified.  In  addition: 

■  DRDC  (and  CAE  PS  through  the  ITS  contract)  can  be  given  free 
licenses.  Negotiation  with  Bohemia  Interactive  will  be  completed  by 
31  March  2010,  and  access  to  the  licenses  should  be  possible  by 
May  2010.  These  licenses  might  include  Bohemia  Fusion  which  is  a 
third-party  tool  creation  package  for  VBS2. 

■  DLSE  can  provide  VBS2  modeling  and  coding  support  to  the  ITS 
project  through  sub-contracts. 

■  The  ITS  project  can  have  access  to  all  VBS2  models  developed  by 
the  CF  (at  least  in  principle)  for  re-purposing  for  the  IEDD  scenario. 
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o  Ngrain.  Ngrain  models  have  been  used  for  mine  clearance.  300  types  of 
mines  have  been  developed  in  Ngrain  for  use  with  a  mobile  device  as  an 
in-field  rehearsal  tool.  These  models  are  unclassified  and  can  be  re¬ 
purposed  for  the  ITS  project  as  required. 

o  Delta  3D.  Delta  3D  is  an  alternative  to  VBS2  that  is  an  open-source  game 
engine  developed  by  the  US  Naval  Postgraduate  School.  The  tool  is 
useful  for  research;  however  the  game  engine  is  older  than  VBS2  and 
requires  specific  expertise  to  code  the  tool. 

o  CAM IX.  The  CAMIX  (Civilian  Modeling  Experimentation)  tool  is  also  a 
promising  approach  that  integrates  with  VBS2  to  model  realistic  civilian 
reactions  to  events  (e.g.,  bomb  blast).  CAMIX  is  a  simulation-independent 
tool  developed  by  DRDC.  Lessons  learnt  from  the  ITS  project  might  be 
useful  for  the  development  of  CAMIX. 

•  Links  with  other  CF  projects  / initiatives.  Participants  discussed  how  DLSE  is 
linked  with  other  CF  projects  and  initiatives.  These  links  are  as  follows: 

o  C-IED  TDP  (DRDC  Toronto).  Collaboration  with  Dr  Jerzy  Jarmasz  and 
Sgt.  Wojtarowicz  regarding  development  of  VBS2  environments  relating  to 
C-IED.  For  example,  the  Environmental  Familiarisation  and  Indicator 
Trainer  (EFIT). 

2.2. 1.6  Directorate  of  Learning  Innovation,  Canadian  Defence  Academy  (CFB 
Kingston) 

The  following  issues  were  discussed  with  LCdr  Tremblay  (Senior  Staff  Officer,  Learning 
Concept  Development)  during  this  stakeholder  meeting: 

•  CDA  Initiatives.  The  following  CDA  initiatives  relevant  to  the  ITS  project  were 
discussed: 

o  Simulation  tools.  Second  Life  and  Open  Second  Life  is  not  currently  in 
favour  due  to  problems  with  the  contracting  vehicle.  Current  plans  are  to 
move  over  to  OpenSim.  OpenSim  is  currently  being  integrated  into  DND 
Learn’s  Learning  Management  System  (LMS)  and  with  content  developed 
for  Second  Life.  Public  Works  have  created  an  Expression  of  Interest  for  a 
government-wide  virtual  world.  VBS2  is  currently  in  the  process  of  being 
purchased  and  a  number  of  air  traffic  control  scenarios  have  been 
developed. 

o  Mobile  Learning.  CDA  conducted  a  literature  review  two  years  ago  to 
examine  the  utility  of  mobile  devices  to  enhance  training  and  performance. 
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o  Physiological  feedback.  CDA  is  conducting  a  technology  watch  on 
biofeedback  for  students. 

•  Links  with  other  CF  projects  / initiatives.  Participants  discussed  how  the  CDA  is 
linked  with  other  CF  projects  and  initiatives.  These  links  are  as  follows: 

o  Institute  of  Information  Technology  (NRC).  Interaction  with  Personal 
Learning  Environment  from  a  social  networking  point  of  view. 

o  Alelo.  CDA  is  investigating  how  Alelo  dialogue  coaching  tools  can  be 
integrated  into  VBS2  for  air  traffic  control  applications. 

•  List  of  Actions.  The  following  actions  were  recorded  during  the  stakeholder 
meeting: 

o  Contact  Directorate  of  Military  Personnel  Operational  Research  and 
Analysis  (DMPORA)  for  ethical  approval  to  collect  physiological  data  from 
CF  personnel. 

2.2. 1.7  Counter-IED  TDP,  Collaborative  Performance  and  Learning  Section 
(DRDC  Toronto) 

The  following  issues  were  discussed  with  Dr  Jarmasz  (Defence  Scientist,  Learning  and 
Training  Group)  and  Sgt  Wojtarowicz  (Project  Coordinator,  I  ED  Awareness  Training 
Project)  during  this  stakeholder  meeting: 

•  IED  Awareness  Technology  Demonstration  Project.  The  following  aspects  of  the 
DRDC-Toronto  IED  TDP  initiatives  relevant  to  the  ITS  project  were  discussed: 

o  Overview  of  TDP.  The  TDP  seeks  to  improve  soldiers’  skills  at  visually 
assessing  IED  threats;  for  example,  identification  of  relevant  cues  and 
assessing  threat  levels.  The  focus  is  on  mounted  operations. 

o  Environmental  Familiarisation  and  Indicator  Trainer  (EFIT).  EFIT  uses 
video  footage  recorded  from  CF  vehicles  in  operations  to  familiarise  CF 
personnel  with  the  routes  and  likely  IED  indicators  along  them  before 
arriving  in  theatre.  A  voice  commentary  and  visual  overlays  on  the  video 
help  to  orient  the  student  to  potential  IED  indicators  along  the  routes  that 
they  will  drive  along  when  deployed.  The  project  has  also  examined  eye 
tracking  data  (e.g.,  scan  patterns  compared  between  experienced  and 
non-experienced  personnel)  and  psychophysiological  measures  (e.g., 
heart  rate  variability  and  galvanic  skin  response). 
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o  VBS2  Training  Videos.  DRDC  is  involved  with  the  development  of  training 
videos  for  the  Combat  Training  Centre  (CFB  Gagetown)  using  content  and 
scenarios  developed  in  VBS2.  The  videos  provide  a  demonstration  of  the 
skills  associated  with  convoys. 

o  Counter  IED  Immersive  Training  Environment  (CUTE).  DRDC  has 

contracted  Chi  Systems  Inc.  to  develop  intelligent  agents  (i.e.,  non-player 
characters)  within  synthetic  environments.  This  project  is  currently  on 
hold;  however  the  CUTE  and  ITS  projects  should  look  for  opportunities  to 
collaborate  in  the  future. 

•  Experiences  using  VBS2.  DRDC  have  used  VBS2  to  create  scenarios  and 
content  pertaining  to  the  training  of  counter-IED  scenarios.  Participants 
discussed  the  lessons  learnt  from  this  experience. 

o  The  VBS  game  engine  is  not  able  to  provide  a  level  of  detail  for  some  IED 
indicators;  such  as  physical  features  of  the  terrain  (e.g.,  terrain  changes, 
ditches,  hills,  elevated  roads),  and  avatars  (restricted  to  walk,  crawl,  shoot 
etc).  Having  said  that,  Bohemia’s  Fusion  development  tool-set  should 
mitigate  most  of  these  issues.  In  addition,  the  CF  has  requested  a  number 
of  changes  to  VBS  in  terms  of  more  Canadian  content  (e.g.,  vehicles, 
small  arms  etc),  more  realistic  vegetation  and  better  road  patches. 

•  Links  with  other  CF  projects  /  initiatives.  Participants  discussed  how  the  Defeat 
the  Device  Centre  of  Excellence  and  the  IEDD  Operator  Course  are  linked  with 
other  CF  projects  and  initiatives.  These  links  are  as  follows: 

o  Combat  Training  Centre  (CFB  Gagetown).  The  CTC  uses,  and  supports, 
the  EFIT  project. 

o  United  States  Research  and  Development  Engineering  Command  (US 

RDECOM).  RDECOM  have  developed  techniques  measure  affective  state 
to  adapt  learning  environments.  For  example,  using  mouse  movements 
and  patterns.  Participants  discussed  the  benefits  of  collaborating  with 
personnel  at  RDECOM  during  the  ITS  project. 


2.2.2  Summary 

In  summary,  the  stakeholder  meetings  were  a  success  insofar  as  they  provided  a  rich 
source  of  information  and  advice  for  the  implementation  of  ITS  technologies,  and  an 
opportunity  to  establish  relationships  with  the  stakeholders  of  the  ITS  project.  The  key 
outcomes  were  as  follows: 
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1 .  The  selection  of  the  IEDD  Operator  Course  for  implementation  of  adaptive 
learning  and  intelligent  tutoring  projects  is  appropriate. 

2.  The  IEDD  Operator  Course  personnel  are  keen  to  support  the  ITS  project  in 
terms  of  access  to  course  materials,  instructors  and  students. 

3.  There  was  a  clear  consensus  that  Second  Life  is  not  an  appropriate 
environment1  (as  we  had  first  thought)  in  terms  of  the  selection  of  the  appropriate 
simulation  and  modeling  tools  to  implement  the  IEDD  Operator  Course  ITS. 
Bohemia  Interactive’s  VBS2  should  be  the  tool  of  choice. 

4.  DLSE  has  offered  to  provide  VBS2  support  to  DRDC  (and  CAE  PS  through  the 
ITS  contract)  in  terms  of  software  licences,  a  smaller  terrain  map,  existing 
content,  and  modeling  and  coding  support  (through  a  sub-contract). 

5.  DRDC  Toronto  (C-IED  TDP)  is  a  key  collaborator  for  the  ITS  project;  both  in 
terms  of  sharing  VBS2  content  pertaining  to  C-IED,  and  identifying  future 
technology  transfer  opportunities  (e.g.,  with  the  CUTE  project). 

The  remainder  of  this  section  describes  an  analysis  of  the  stakeholders  and  analyses  of 
the  IEED  Operator  course  in  order  to  inform  the  implementation  and  evaluation  plans. 

2.3  Stakeholder  Analysis 

This  section  summarizes  the  results  of  the  stakeholder  analysis,  with  particular 
emphasis  on  presenting  results  of  the  analysis  of  the  IEDD  Operator  Course. 


2.3.1  Methodology 

The  stakeholder  analysis  was  conducted  by  collecting  data  during  individual  meetings 
with  stakeholders.  Stakeholders  were  identified  through  DRDC  and  other  CF  contacts  in 
the  statement  of  work.  As  work  progressed  additional  stakeholders  were  identified  and 
added  to  meeting  agendas  as  possible.  Stakeholder  data  were  collected  and 
summarized  using  a  generic  format  built  from  CAE  PS  change  management  and 
communication  planning  best  practices.  A  prioritization  scheme  has  been  identified 
using  a  common  change  management  /  stakeholder  analysis  power  and  involvement 


1  In  addition  to  problems  associated  with  licensing  and  contractual  arrangements  with  Linden  Labs,  Second  Life  is  not  targeted  at  training  or  at 
re-use  of  standards  based  source  data.  There  are  no  mechanisms  for  customization  of  the  Second  Life  capability.  There  is  a  permissions 
structure  to  control  ‘visitor’  access  to  land  under  the  user’s  ownership,  but  the  Second  Life  space  is  essentially  public.  Investigation  has 
revealed  that  Linden  Labs  recommends  40  users  as  the  peak  concurrent  user  density  for  a  region  of  land  (Unrau  et  al.,  1 999).  While  Second 
Life  supports  a  large  number  of  users,  it  does  not  support  a  high  density  of  users.  Finally,  given  that  Second  Life  is  a  virtual  world  (as  opposed 
to  a  virtual  environments),  students  of  the  IEDD  Operator  Course  would  need  to  schedule  time  within  Second  Life  for  them  to  be  able  to  use 
the  ITS  training  scenario. 
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frame  presented  using  a  structure  built  on  MindTools™,  power  interest  analysis  for 
stakeholder  prioritization  (www.mindtools.com/rs/StakeholderAnalvsis). 

Stakeholders  were  identified  using  the  following  criteria: 

1 .  Someone  who  is  impacted  by  the  project;  and 

2.  Someone  who  can  have  an  impact  (positive  or  negative),  on  the  project. 

Stakeholders  can  be  individuals,  either  identified  by  their  job  title  or  by  the  actual  name, 
or  they  can  be  groups  of  individuals  such  as  department  or  teams.  They  can  also  be 
organisations,  approval  bodies,  and  so  on.  This  analysis  identified  eight  stakeholder 
groups  -  all  organizations.  The  nature  of  the  data  collected  for  each  organization  is 
described  below  (details  and  data  are  included  in  Appendix  A): 

•  Stakeholder  organisation.  Understanding  the  stakeholder  organization  is 
important.  The  analysis  provides  the  name  and  location  of  eight  stakeholder 
organizations.  This  information  should  be  reviewed  yearly  to  ensure  the  ITS 
project  maintains  knowledge  of  changing  titles,  mandates  and  activities  inside 
and  outside  of  DND. 

•  Stakeholder  name  and  location.  The  name  and  location  of  the  individual 
identified  as  the  ITS  project’s  primary  point  of  contact  is  included  in  the  analysis. 
Where  possible  a  second  point  of  contact  is  also  identified. 

•  Stakeholder  role.  The  function,  job  and  domain  of  responsibility  of  the  individual 
stakeholder  are  included  in  the  stakeholder  analysis.  This  information  helps 
identify  the  individual’s  interest,  level  of  influence  and  provides  a  high  level  view 
of  where  the  individual  fits  inside  their  organization. 

•  Stake  /  interest  in  the  ITS  Project.  Based  upon  meeting  outcomes,  as  well  as  the 
ITS  team  and  DRDC  client’s  knowledge  and  experience  of  the  stakeholders 
groups  and  individuals,  the  analysis  identifies  why  this  person  would  be 
interested  in  this  project.  These  data  were  assessed  through  meeting  outcomes 
and  notes  taken  during  meeting  discussions.  It  identifies  what  stake  the  individual 
has  in  the  project  by  asking  if  it  affects  their  job,  their  role,  their  perceived 
position  within  an  organisational  structure.  This  information  assists  in  determining 
how  the  amount  and  type  of  communication  the  stakeholder  could  receive  to 
maintain  their  interest  and  support. 

•  ITS  related  programs  and  program  links.  Data  collected  in  the  meetings  identified 
stakeholder  activities  or  projects  that  had  related  themes,  capabilities  or  content. 
This  included  experience  with  technologies  in  evaluation  to  support  elements  of 
the  intelligent  tutoring  and  adaptive  learning  goals  of  the  project.  This  information 
highlights  related  projects  and  programs  that  might  lead  to  identification  of 
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missing  in  stakeholders.  It  increases  the  project’s  knowledge  of  related  projects 
and  capabilities  that  may  be  useful  align  with  and  ensure  activities  are  not 
duplicated  but  coordinated  and  aligned  for  a  larger  impact.  This  data  also  helps 
identify  projects  or  individuals  who  act  as  central  ‘nodes’  for  information  and 
knowledge  in  the  research  domain  and  related  training  or  distance  learning 
domains. 

•  ITS  related  capabilities.  Meeting  discussions  informally  identified  the  core 
activities  and  capabilities  of  the  stakeholder  groups.  These  capabilities  were 
noted  as  either:  capacity  and  knowledge  of  the  people  in  the  organization; 
processes  that  might  be  useful  to  leverage  for  the  next  phases  of  the  project  or, 
availability  of  knowledge  of  technologies.  Capturing  this  data  early  in  the  ITS 
project  enables  the  ITS  project  team  and  its  partners  to  leverage  the  right 
stakeholders  at  the  time,  to  solicit  cooperation  and  buy-in  for  the  project  and 
identify  overall  efficiencies  that  could  be  achieved  by  involving  stakeholder  in 
various  phases  of  the  project. 

•  Role  in  phase  of  project  involvement.  The  implementation  plan  identifies  three 
phases  of  the  ITS  project:  Analysis  and  Design,  Implementation  and  Evaluation. 
Details  of  the  timing  and  scope  of  activities  in  each  of  these  phases  are  included 
in  this  report.  The  analysis  reviews  stakeholder  interest,  power,  capabilities  as 
discussed  above  to  identify  how  best  to  involve  the  stakeholder  in  future  phases 
of  the  project.  Specific  types  of  involvement  will  depend  on  the  phase  and  the 
stakeholder.  These  are  identified  in  the  stakeholder  needs  section  below. 

•  Stakeholder  needs.  In  light  of  the  information  collected  and  analysis  performed 
above,  this  section  recommends  what  the  stakeholder  needs  from  the  ITS  team 
and  what  the  ITS  team  could  benefit  from  the  stakeholder.  Needs  are 
characterized  into  four  types: 

o  Communication  needs; 

o  Information  provision; 

o  Advise  and  support;  and 

o  Inclusion  in  project  team  improvement  activities. 

•  Perceived  attitude  and  associated  risks.  Based  upon  the  CAE  PS  team's 
experience  dealing  with  this  individual  in  the  stakeholder  meetings,  the  analysis 
provides  an  initial  assessment  of  the  stakeholder's  attitude  towards  the  ITS 
project. 

•  Project  impact  on  stakeholder.  The  project  impact  on  the  stakeholder  provides  an 
indication  of  the  priority,  involvement  and  influence  the  stakeholder  may  have  on 
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the  project.  A  three  level  scale  of  high,  medium,  low  was  used  to  assess  the  ITS 
project’s  impact  on  the  stakeholder: 

o  High:  The  stakeholder  is  significantly  impacted  by  the  outcome  of  this 
project. 

o  Medium:  There  will  be  some  impact  on  the  stakeholder  from  the  outcome 
of  this  project. 

o  Low:  There  will  be  little  or  no  impact  to  stakeholder  from  the  outcome  of 
this  project. 

•  Connection  of  other  stakeholders.  Due  to  overlapping  capabilities  and 
relationships  between  organizations,  this  section  of  the  analysis  identifies  a 
stakeholder’s  connection  to  other  stakeholders  included  in  this  analysis.  These 
data  provide  a  high  level  view  of  direct  and  indirect  relationships  between 
stakeholders  as  well  as  linkages  to  corporate  wide  agencies  and  organizations 
such  as  Canadian  Defence  Academy  (CDA),  Defence  Research  Development 
Canada  (DRDC)  and  the  National  Research  Council  (NRC). 

2.3.2  Results 

Meetings  took  place  at  the  place  of  work  of  the  eight  stakeholder  groups  and  their  points 
of  contact.  In  some  cases,  multiple  points  of  contact  were  available  to  participate  and 
contribute  to  the  stakeholder  meetings.  The  meetings  were  conducted  using  an  informal 
approach  designed  to  collect  a  wide  range  of  information.  The  approach  and  outcome  of 
these  meetings  are  summarized  in  Section  2.2  of  this  report.  The  outcomes  of  the 
meeting  that  provided  information  into  the  stakeholder  analysis  were  organized  into 
eleven  types  of  information  as  described  in  the  methodology  section  above.  The  results 
of  this  data  provide  an  overall  picture  of  the  stakeholders  power,  influence,  interest  and 
potential  involvement  in  future  phases  of  the  ITS  project.  The  stakeholders  identified 
can  be  categorized  as  follows: 

•  1  x  Army  Simulation  and  Technology  Stakeholder; 

•  3  x  Subject  Matter  Experts  in  Army  Training  and  Counter-IED; 

•  2  x  DRDC  Stakeholders; 

•  1  x  DND  Future  Training  Organization;  and, 

•  1  x  Other  Government  Department 


28  May  201 0  -  23  -  5045-1  Version  02 

©  Her  Majesty  the  Queen  as  represented  by  the  Minister  of  National  Defence,  2010 
©  Sa  majeste  la  reine,  representee  par  le  ministre  de  la  Defense  nationale,  2010 


CAE 


Stakeholder  Analysis  for  C-IED  T raining  Courses 


2.3.3  Recommendations 

The  implementation  plan  and  conduct  of  the  project  must  be  cognisant  of  the  roles  of 
stakeholders  during  the  different  project  phases.  Areas  for  consideration  in  the  detailed 
implementation  plan  are  organized  by  project  phase: 

•  Analysis  and  design  phase.  Identify  areas  where  capabilities  of  stakeholders 
might  afford  the  ITS  project  an  advantage.  During  the  analysis  and  design  phase, 
the  role  of  subject  matter  experts  is  particularly  important  and  demands  a  higher 
level  of  involvement  than  during  other  phases.  Communication  and  buy-in  for 
stakeholders  who  may  provide  subject  matter  knowledge  will  be  key  during  this 
phase. 

•  Implementation  phase.  Identify  areas  where  capabilities  of  stakeholders  may 
provide  the  ITS  with  an  advantage.  For  instance,  are  there  people  in  stakeholder 
organizations  with  the  skills  or  knowledge  to  do  reviews  or  contribute  media 
content  or  technologies  to  the  project? 

•  Evaluation  phase.  Identify  appropriate  stakeholders  to  participate  in  the 
evaluation  of  the  project  outcome,  and  stakeholders  who  can  create  a  larger 
degree  of  support  and  information  dissemination  about  the  results  of  the  ITS 
project. 

Finally,  it  is  recommended  that  the  project  team  maintains  a  ‘technology  watch’ 
throughout  the  duration  of  the  project.  The  technology  watch  will  include  activities 
pertaining  to: 

•  Monitoring  of  virtual  world  and  gaming  technologies; 

•  Monitoring  role  changes  in  the  stakeholder  community; 

•  Monitoring  for  new  stakeholders; 

•  Maintaining  stakeholder  involvement  at  the  right  time; 

•  Communicating  project  outcomes  with  stakeholders  and  receiving  any  new 
findings  or  knowledge  of  the  stakeholders  themselves; 

•  Providing  feedback  loops  as  a  way  to  keep  stakeholders  involved  in  the  project; 
and, 

•  Ensuring  that  project  deliverables  are  visible  to  the  appropriate  stakeholder  (to  be 
determined  on  a  case-by-case  basis). 
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2.4  IEDD  Operator  Course  ITS  Gap  Analysis 

The  following  section  describes  the  IEDD  Operator  Course  in  terms  of  a  general  course 
overview,  participants  and  the  delivery  methods  employed.  The  objective  of  this  section 
is  to  identify  components  of  the  IEDD  Operator  Course  that  would  benefit  most  from  the 
implementation  of  adaptive  learning  and  intelligent  tutoring  technologies.  This  work 
builds  upon  the  analysis  conducted  by  Banbury  et  al.  (2008). 

2.4.1  IEDD  Operator  Course  Analysis 

This  course  enables  CF  personnel  to  identify,  disrupt,  and  dispose  of  lEDs.  Personnel 
will  also  be  able  to  identify,  recognize  and  formulate  an  accurate  threat  assessment  of 
the  suspected  IED,  and  provide  advice  on  immediate  protective  measures  against 
hazards  associated  with  chemical,  biological,  and  radiological  (CBR)  improvised 
devices.  This  course  excludes  yield-producing  nuclear  weapons  and  devices. 


2.4.1. 1  IEDD  Operator  Course  Overview 

An  IEDD  team  comprises  of  a  Number  1 ,  who  fulfills  the  role  of  team  leader,  and  a 
Number  2,  who  fulfills  the  role  of  an  assistant  to  the  Number  1  and  operates  the  IEDD 
equipment  (e.g.,  video,  robot).  Typically,  the  IEDD  teams  are  called  out  to  an  incident 
involving  a  suspected  (or  exploded)  IED  or  explosive  device  that  has  been  discovered 
and  are  responsible  for  the  safe  disposal  of  the  device.  On  arrival,  the  IEDD  team  sets 
up  an  ad  hoc  command  post  from  a  safe  position  where  the  robot  can  be  controlled.  In 
order  to  successfully  dispose  of  the  explosive  device,  it  is  critical  that  the  IEDD  team 
accurately  identifies  the  type  of  device  so  that  the  correct  Render  Safe  Procedures 
(RSPs)  can  be  used.  The  threat  assessment  process  involves  both  the  close  inspection 
of  the  device  itself  and  the  gathering  of  information  from  third  parties  (e.g.,  civilians, 
local  police,  CF  personnel  etc)  through  questioning  to  determine  the  level  of  threat 
posed  to  personnel  by  the  device.  Any  information  gleaned  from  these  activities  (e.g., 
intelligence  about  the  type  of  devices  used)  is  reported  to  Headquarters. 

The  IEED  Operator  Course  is  based  on  the  Qualification  Standard  (QS)  Improvised 
Explosive  Device  Disposal  (IEDD)  Operator  (reference:  A-P3-002-IED/PH-H01).  The 
IEED  Operator  Course  is  aimed  at  qualifying  students  for  the  role  of  Number  1 .  All 
students  on  this  course  would  have  already  qualified  for  the  Number  2  role.  In  addition, 
the  course  enrolment  also  includes  existing  Number  1  qualified  personnel  who  require 
re-qualification  (the  qualification  is  valid  for  three  years). 

The  following  information  was  acquired  over  the  course  of  the  Stakeholder  Analysis 
interviews  and  from  previous  discussions  with  Maj  Schurman  (NDHQ)  conducted  under 
the  previous  contract  (Banbury  et  al.,  2009). 
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•  General  course  information.  General  course  information  was  discussed  as  part  of 
the  stakeholder  analysis  meetings: 

o  Course  Participants.  Students  come  from  a  broad  range  of  backgrounds 
(e.g.,  AVN  techs,  Navy  clearance  divers,  Ammo  techs  and  combat 
engineers),  with  a  broad  mix  of  skills,  rank  and  age.  However,  most 
students  have  a  typical  Type  A’  personality. 

o  Course  Schedule.  There  are  three  courses  run  per  year  (i.e.,  spring, 
summer  and  fall)  with  approximately  20  students  per  course. 

o  Course  Success.  At  present,  there  is  a  40  percent  failure  rate  due  to 
deficiencies  in  skills  associated  with  threat  assessment.  What  those 
deficiencies  actually  are  is  not  understood;  however,  the  course 
instructors  speculate  this  it  is  due  to  poor  Questioning  Technique  skills. 

•  Course  phases.  The  IEDD  Operator  Course  is  divided  into  three  phases: 

o  General  knowledge.  Instruction  pertaining  to  three  classes  of  lEDs  (i.e., 
command-operated,  timer-operated  and  victim-operated  devices),  and 
what  components  are  used  to  construct  /  improvise  these  devices. 

o  Skills.  Instruction  pertaining  to  threat  assessment  (which  includes 
Questioning  Technique),  developing  a  plan  based  on  the  threat 
assessment,  and  the  execution  of  that  plan  (i.e.,  following  the  appropriate 
RSPs).  The  Questioning  Technique  is  by  far  the  most  difficult  component 
of  the  course,  and  is  currently  taught  using  a  combination  of  PowerPoint 
slides  (see  Figure  2-1)  and  classroom  role-playing  exercises.  It  was 
suggested  by  the  IEDD  Operator  Course  stakeholders  that  the  ITS  project 
concentrate  on  this  aspect  of  the  course,  as  it  should  have  a  significant 
impact  on  the  failure  rate.  In  addition,  the  disposal  plan  might  have  to  be 
adapted  in  light  of  new  information  discovered  during  the  execution  of  the 
RSPs.  However,  many  operators  are  not  flexible  and  reach  premature 
(incorrect)  conclusions  from  partial  data,  and  do  not  change  the  plan  in 
response  to  new  data  (i.e.,  confirmation  bias).  As  such,  it  was  also 
suggested  that  the  ITS  project  concentrate  on  this  aspect  of  the  course  as 
well,  as  it  should  have  a  significant  impact  on  the  failure  rate  also. 

o  Scenario-based  testing.  Students  receive  nine  scenarios  in  total.  Six  are 
practice  scenarios  (i.e.,  two  command-operated  scenarios,  two  timer¬ 
operated  scenarios,  and  two  victim-operated  scenarios),  and  three  are 
evaluation  scenarios  (i.e.,  one  for  each  of  the  three  classes  of  device).  For 
the  evaluation  scenarios,  students  are  divided  into  five  teams  of  two 
students,  and  all  teams  receive  the  same  scenario.  Instructors  play  the 
role  of  witnesses  to  be  questioning  during  the  threat  assessment  phase, 
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and  provide  pre-determined  responses  to  questions  from  students  relating 
to  the  main  teaching  points  (MTPs).  The  success  of  each  student  is 
determined  by  the  student  getting  90  percent  of  the  available  information 
from  their  questioning.  Ideally,  students  should  start  to  plan  and  execute 
the  RSPs  with  40  to  50  percent  of  the  information  and  then  re-plan  (or  not) 
when  conducting  the  disposal  and  more  information  comes  to  light. 
Nevertheless,  there  are  20  to  25  questions  that  always  need  to  be  asked. 
These  questions  are  broadly  described  as  the  “5  Ws”  -  what,  when, 
where,  why  and  who.  However,  there  is  not  a  checklist  of  questions;  nor 
should  there  be.  Rather,  students  are  taught  questioning  technique,  rather 
than  what  questions  should  be  asked. 
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'A  Stage  One  -  Questioning  Technique 

(  A  Stage  One  -  Questioning  Technique 

Open  Ended  Questions 

Open  questions  have  the  following  characteristics 

•  They  ask  the  respondentto  think  and  reflect 

•  They  will  give  you  opinions  and  thoughts 

•  They  can  sometimes  hand  control  of  the  conversation  to 
the  respondent 

•  Encourage  an  individual  to  talk  and  provide  maximum 
information  to  identify  causes  and  work  toward  solutions. 

•  Usually  begin  with  "how",  "why"  or  "could". 

•  Example  "Could you  descnbe  the  item  in  detail?" 

Leading  Questions 

•  Have  their  place,  but  not  if  you  wantto  get 
unbiased  answers.  Think  carefully  about  how 
the  other  person  may  interpret  the  question, 
i.e.  “so  the  wires  just  went  into  the  ground ?" 
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\  A  Stage  One  -  Questioning  Technique 

Avoid  Jargon 

•  Jargon  is  useful  for  people  who  specialize  in 
the  same  subject  as  it  allows  them  to  talk  in 
'shorthand'.  It  is  sometimes  useful  to  use  it 
with  other  people  to  signal  your  expertise. 
Mostofthe  time,  however,  it  just  annoys  and 
confuses  other  people. 

•  Avoid  complex  language 

Silence 

•  A  non-verbal  probe  can  also  be  used,  for 
example  by  raising  your  eyebrows  and  tilting 
yourhead.  This  shows  youare  interested  in  a 
particularpointand  they  may  give  you  more 
detail  withoutyou  having  to  say  anything 

Figure  2-1:  Example  of  Questioning  Technique  Course  Content 
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2.4.1. 2  Recommendations 

This  section  provides  a  list  of  recommendations  for  tasks  to  be  conducted  early  during 
the  next  phase  of  the  project  in  order  to  support  the  implementation  of  ITS  technologies 
within  the  IEDD  Operator  Course.  Specifically: 

1 .  Conduct  literature  review  on  Questioning  Technique.  In  order  to  create  new 
instructional  material  as  part  of  the  ITS  scenario,  a  literature  review  of 
questioning  techniques  needs  to  be  undertaken.  The  review  should  explore 
questioning  and  interviewing  techniques  used  in  qualitative  psychology,  doctor- 
patient  interviews,  police  investigations  (e.g.,  National  Investigation  Services). 
The  review  should  also  include  cognitive  biases  that  are  thought  to  affect  threat 
assessment  and  decision  making  (e.g.,  confirmation  bias,  cognitive  tunnelling), 
and  strategies  by  which  questioners  can  adopt  in  order  to  mitigate  them  (e.g., 
problem  solving  and  decision  making  techniques  taught  as  part  of  aviation  Crew 
Resource  Management).  The  intention  is  for  these  biases,  and  means  to  mitigate 
them,  will  be  explored  within  the  auspices  of  the  ITS  scenario.  The  review  should 
also  include  any  lessons  learnt  from  the  meeting  with  Maj  Terfry  at  the  School  of 
Military  Intelligence  (CFB  Kingston)  concerning  the  Tactical  Ouestioning  course, 
and  Dr  Bruce  Spencer  at  the  Institute  for  Information  Technology  (NRC, 
Fredericton)  regarding  the  adaptive  learning  system  for  doctor-patient  interviews. 
Finally,  the  review  should  also  seek  to  identify  the  causes  of  failure  of  IEDD 
Operator  Course  students.  Correlates  to  be  examined  should  include  existing 
psychometric  data  collected  during  enrolment  into  the  CF.  This  work  should  be 
completed  in  collaboration  with  the  relevant  personnel  at  NDHO  who  would 
provide  the  necessary  data. 

2.  Organise  scenario  development  and  instructional  design  workshop.  This 
workshop  will  support  the  development  of  a  suitable  mission  scenario  for  the 
implementation  of  ITS-related  technologies  into  the  IEDD  Operator  Course.  This 
should  be  held  in  May  2010  at  the  School  of  Military  Engineering,  CFB 
Gagetown.  This  workshop  will  also  develop  ITS  training  and  evaluation  scenarios 
based  on  the  mission  scenario.  Specifically,  Main  Teaching  Points  (MTPs)  will  be 
developed  for  the  following  Performance  Objectives  (POs)  from  the  Oualification 
Standard  (OS)  Improvised  Explosive  Device  (IEDD)  Operator  (reference:  A-P3- 
002-IED/PH-H01):  PO  5.10  Good  Ouestioning  Technique,  and  PO  5.1 1  Task 
Appreciation  /  Threat  Assessment.  Finally,  evaluation  criteria  for  these  POs  will 
also  be  developed. 

Section  3.4.4  describes  how  these  tasks  fit  into  the  implementation  schedule  for  the 
next  phase  of  the  project. 


28  May  201 0  -  28  -  5045-1  Version  02 

©  Her  Majesty  the  Queen  as  represented  by  the  Minister  of  National  Defence,  2010 
©  Sa  majeste  la  reine,  representee  par  le  ministre  de  la  Defense  nationale,  2010 


CAE 


Stakeholder  Analysis  for  C-IED  T raining  Courses 


3  IMPLEMENTATION  PLAN 

This  section  of  the  report  describes  the  implementation  plan  for  the  IEDD  Operator 
Course  ITS.  The  first  part  describes  the  implementation  of  the  ITS  in  terms  of  its  content 
and  capabilities,  and  then  describes  the  architecture  of  the  ITS  in  more  detail.  The  final 
part  of  this  section  describes  the  schedule  for  the  design,  implementation  and 
evaluation  of  the  ITS. 


3.1  Background 

Before  describing  the  content  and  capabilities  of  the  IEDD  Operator  Course  ITS,  this 
section  reviews  the  results  of  parallel  activities  undertaken  to  review  and  identify 
adaptation  mechanisms  for  adaptive  learning  and  intelligent  tutoring  technologies  (see 
Kramer,  Tryan  and  Banbury;  2010). 

Kramer  and  colleagues’  report  reviewed  current  techniques  and  technologies  relating  to 
eye-tracking,  psychophysiological  indices  of  workload  and  stress,  learning  styles  and 
performance  tracking  that  have  been  successfully  used,  or  could  theoretically  be  used, 
to  augment  the  IEDD  ITS.  The  purpose  of  the  review  was  to  provide  straightforward, 
applicable  recommendations  that  would  lead  to  the  purchase  of  technology  that  can  be 
implemented  into  the  IEDD  ITS.  The  findings  from  this  review  are  summarised  in  the 
following  sections. 


3.1.1  Eye  Tracking-based  Adaptation 

Eye-tracking  as  a  means  of  attention,  workload,  and  expertise  proved  to  be  the  most 
mature  and  commercially  accessible  technologies  used  in  research.  The  recommended 
style  of  eye-tracker  is  a  desktop,  or  screen-integrated  eye-tracker,  as  they  are  the  most 
non-invasive  and  should  take  the  least  amount  of  time  to  setup  and  calibrate  as 
compared  to  a  head  mounted  eye-trackers.  Goggle  style  eye-trackers  are  appealing  for 
their  mobility,  but  are  unnecessary  and  costly  for  use  in  front  of  a  PC.  The  main  limiting 
factor  is  price,  as  the  higher  quality  and  more  frequently  used  models  in  peer-reviewed 
research  tend  to  cost  over  $40,000.  The  suggested  quality/price  compromise  is  to 
purchase  one  mid-range  ($10,000  or  less)  eye-tracker  (such  as  the  EasyGaze  system) 
to  use  to  collect  baseline  data  and  to  calibrate  the  other  systems.  If  numerous  eye- 
trackers  are  needed  to  be  left  at  the  IEDD  course,  it  is  recommended  to  use  open- 
source,  or  web-cam  based  alternatives  which  could  be  calibrated  against  the  higher 
quality  version  purchased. 
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3.1.2  Psychophysiological-based  Adaptation 

The  psychophysiological  measure  recommended  for  use  is  Heart  Rate  Variability  (HRV) 
in  either  a  simple  ear-clip,  dry-electrode  wrist  straps,  or  wireless  Bluetooth  model.  HRV 
uses  non-invasive  techniques  for  assessment,  and  could  be  used  in  combination  with 
eye-tracking  to  help  triangulate  a  more  accurate  meaning  of  general  arousal  to  specific 
stimuli.  The  technology  recommended  for  purchase  HRV  Live!  from  the  company 
Biocom. 

EEG,  while  a  valuable  and  proven  source  of  mental  workload  and  other  cognitive 
functions,  is  not  recommended  as  a  form  of  adaptive  measure  within  the  current  ITS. 
First  and  foremost,  the  equipment  and  setup  time  required  to  acquire  accurate  EEG 
recordings  makes  it  unlikely  to  be  usable  by  IEDD  operators  without  assistant  or 
researcher  supervision.  The  ITS  needs  to  be  a  standalone  learning  tool  that  is  readily 
accessible  by  students.  If  the  ITS  requires  an  hour  for  placing  gels  and  electrodes  on 
one’s  scalp,  it  is  doubtful  that  it  will  be  used  for  more  than  purely  research  purposes 
when  the  experimenter  is  present. 


3.1.3  Learning  Style-based  Adaptation 

The  Index  of  Learning  Styles  (Felder  and  Solomon,  2006)  is  recommended  to  be  given 
to  students  prior  to  using  the  IEDD  ITS.  The  ILS  has  proven  validity  and  reliability,  is 
freely  available  and  automatically  scored  online,  and  takes  only  a  few  minutes  to 
complete.  It  is  further  recommended  to  administer  the  index  during  baseline  testing  in 
order  to  get  a  sense  of  any  dominant  learning  styles.  These  results  would  help  to 
provide  guidance  for  content  building  within  the  actual  ITS;  such  as  a  preference  for 
visual  over  auditory  learning  styles. 


3.1.4  Attention  Tracking-based  Adaptation 

Performance  tracking  measures  produced  the  least  amount  of  tangible  technologies 
that  could  be  implemented  into  the  IEDD  ITS.  While  motion  tracking  offered  some 
interesting  research  streams  for  performance  tracking  based  on  postures,  at  this  point, 
the  related  technologies  are  only  academic  prototypes.  The  most  plausible  form  of 
performance  tracking  appears  to  be  speech  recognition.  This  has  matured  in  the  field  of 
language  training  as  in  Rosetta  stone,  but  is  still  in  need  of  improvement  for  Alelo.  The 
problem  is  that  the  project  is  not  in  need  of  a  language  training  just  yet,  so  this 
technology  is  likely  too  complex  and  not  reliable  enough  to  use  yet.  As  for  Alelo,  the 
style  of  their  interactive,  as  well  as  mobile,  training  is  certainly  appealing  -  the  question 
is  if  the  program  itself  can  be  easily  integrated. 
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3.2  Recommended  Implementation  of  ITS  Technologies  into  the 
IEDD  Operator  Course 

The  stakeholder  meetings  discussed  the  potential  of  adding  an  additional  practice 
scenario  to  the  IEDD  Operator  Course  before  the  evaluations.  This  scenario  would 
provide  the  ITS  project  team  the  opportunity  to  evaluate  the  impact  of  adaptive  learning 
and  intelligent  tutoring  technologies  within  the  IEDD  Operator  Course.  Specifically,  the 
evaluation  would  both:  (i)  afford  the  ITS  project  team  with  an  opportunity  to  fully- 
exercise  a  range  of  adaptive  learning  and  intelligent  tutoring  technologies,  and  (ii) 
provide  the  greatest  possible  benefit  to  students  on  the  IEDD  Operator  Course.  This 
section  presents  a  notional  IEDD  Operator  Course  ITS  scenario  and  notional 
instructional  design  based  on  it. 


3.2.1  Notional  IEDD  Operator  Course  ITS  Scenario 

The  interpretation  of  the  rules  and  procedures  rely  heavily  on  the  situation  assessment 
and  decision-making  activities  undertaken  in  the  field  under  considerable  time  pressure 
and  stress.  As  such,  students  are  often  prone  to  decision-making  biases.  For  example, 
previous  academic  research  has  shown  that  when  people  seek  data  to  test  their  beliefs, 
they  often  select  data  that  serves  to  confirm  them  (e.g.  Confirmation  Bias;  Wason, 

1960).  Furthermore,  despite  having  information  that  contradicts  such  beliefs,  people  are 
often  very  slow  to  change  their  minds  (e.g.  Belief  Perseverance;  Ross,  Lepper  and 
Hubbard,  1975). 

Ideally,  students  will  use  the  IEDD  ITS  on  an  individual  and  self-paced  basis,  in  order 
for  the  ITS  team  to  evaluate  its  utility  for  distance-learning  applications.  For  example, 
the  IEDD  ITS  can  be  used  during  the  course  itself,  or  as  refresher  training  during 
exercises  such  as  those  pertaining  to  the  ‘Road  to  High  Readiness”.  The  requirements 
for  the  IEDD  ITS  scenario  were  discussed  during  the  stakeholder  meetings  and  were  as 
follows: 

•  The  scenario  should  be  non-permissive.  In  other  words,  students  should  be 
under  some  degree  of  time  pressure  to  conduct  their  threat  assessment  and  plan 
their  disposal  strategies; 

•  The  scenario  should  engender  high  levels  of  workload  and  pose  a  significant 
challenge  to  students  in  terms  of  the  complexity  of  the  threat  assessment 
required  in  order  to  determine  the  correct  RSP; 

•  The  information  available  within  the  scenario  (i.e. ,  Situation  Awareness 
‘elements’)  should  be,  on  the  surface,  conflicting  and  ambiguous.  However, 
closer  inspection  of  these  elements  should  resolve  any  conflicts  and  ambiguities. 
The  intention  is  to  promote  a  situation  in  which,  on  the  surface,  one  particular 
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course  of  action  is  evident.  However,  if  the  student  digs  a  little  deeper  into  the 
information,  contradictory  information  is  found  which  should  change  their  planned 
course  of  action.  In  other  words,  we  seek  to  promote  situations  in  which  students 
might  demonstrate  ‘confirmation  bias’  which  affects  their  ability  to  change  their 
planned  course  of  action  in  light  of  contradictory  information. 

•  The  scenario  needs  to  be  realistic  and  relevant  to  current  operational  realities 
and  complexities. 

The  content  of  the  scenario  itself  was  discussed,  and  the  following  section  lists 
suggestions  for  its  content: 

•  The  scenario  should  be  post-blast  with  a  secondary  device  (victim-operated). 

The  blast  involved  a  CF  convoy.  No  damage  to  vehicles  or  injuries  to  personnel 
were  sustained. 

•  The  following  Non-Player  Characters  (NPCs)  should  be  present: 

o  CF  witnesses.  Students  will  interact  with  the  CF  personnel  that  observed 
the  post-blast  scene  when  conducting  a  dis-mounted  patrol  following  the 
primary  blast.  These  witnesses  would  have  observed  critical  SA  elements 
that  the  student  must  find  out  about.  Ideally,  all  witnesses  should  be  kept 
apart  to  ensure  the  objectivity  of  the  witnesses. 

o  Afghan  National  Army  (ANA)  or  Police  (ANP)  witness.  Students  will 

interact  with  the  ANA/ANP  officer  in  charge  who  will  speak  for  the  ANA  or 
ANP  witness.  Students  should  also  be  cognisant  of  cultural  sensitivities 
when  interacting  with  these  types  of  NPCs.  For  example,  some  cultures 
might  not  want  to  readily  admit  to  an  authority  figure  that  they  do  not  know 
something,  and  will  instead  try  to  be  helpful  by  being  ‘creative’  with  their 
answers.  Again,  this  witness  would  have  observed  critical  SA  elements 
that  the  student  must  find  out  about.  Finally,  the  implementation  of  the 
ANA  or  ANP  witness  should  assume  that  the  witness  can  speak  English 
(or  French)  and  does  not  require  an  interpreter  NPC. 

o  Number  2  IEDD  operator.  The  Number  2  operator  will  act  as  an  assistant 
to  the  Number  1 ,  but  will  also  be  qualified  as  a  Number  1 ;  current 
operational  practice  is  that  IEDD  operators  will  take  it  in  turns  to  act  as 
Number  1  when  responding  to  IED  incidents.  As  such,  the  Number  2  can 
be  a  source  of  information  and  advice.  In  terms  of  the  ITS  project,  the 
Number  2  could  be  a  source  of  hinting  and  coaching  for  the  student.  For 
example,  the  Number  2  can  remind  the  Number  1  of  critical  SA  elements. 

o  Quick  Response  Force  (QRP)  personnel.  Students  will  not  necessarily 
need  to  interact  with  these  personnel  as  part  of  their  questioning,  but  they 
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should  be  present  in  the  simulation.  These  NPCs  could  be  used  to 
manipulate  the  time  pressure  of  the  scenario. 

o  On  Scene  Commander  (OSC).  The  OSC  will  be  the  first  point  of  contact 
with  the  CF  patrol  on  arrival  at  the  scene  of  the  post-blast.  The  OSC  will 
brief  the  student  about  the  situation  and  identify  the  witnesses  for  the 
student  to  question.  The  OSC  is  also  responsible  for  coordinating  the 
security  cordon  to  protect  both  his/her  personnel  and  those  of  the  QRF 
and  IEDD  team.  Similar  to  the  QRF  commander,  the  OSC  can  also  be 
used  to  manipulate  the  time  pressure  of  the  scenario. 


3.2.2  Notional  IEDD  Operator  Course  ITS  Instructional  Design 

At  present,  the  teaching  of  questioning  technique  (QT)  consists  of  PowerPoint-based 
instruction  on  the  knowledge  and  procedures  relating  to  the  determination  of 
appropriate  courses  of  action  when  disposing  of  lEDs  and  explosives.  Course 
evaluation  activities  comprise  of  testing  the  students’  retention  of  this  knowledge  using 
role-played  evaluations  with  instructors. 

Competence  is  often  broken  down  into  Knowledge,  Skills  and  Attitudes  (KSAs).  The 
KSAs  pertaining  to  the  notional  IEDD  ITS  scenario  are  as  follows: 

•  Knowledge.  Understanding  of  what  factors  need  to  be  observed  in  order  to 
classify  the  suspected  device  as  either  a  command,  timing  or  victim-operated 
device. 

•  Skills.  Questioning  technique  of  military  and  civilian  witnesses. 

•  Attitudes.  The  personality  trait  of  ‘conscientiousness’.  This  personality  trait 
includes  such  elements  as  self-discipline,  carefulness,  thoroughness, 
organization,  deliberation  (i.e.,  the  tendency  to  think  carefully  before  acting),  and 
need  for  achievement. 

Although  we  seek  to  improve  the  QT  of  students,  it  must  be  done  within  the  context  of 
the  competencies  required  to  execute  the  entire  mission.  For  example,  the  effectiveness 
of  the  QT  skills  of  the  student  will  be  a  function  of  the  knowledge  of  how  that  information 
should  be  used  to  assess  the  level  of  threat,  and  the  conscientiousness  of  the  student 
to  persevere  with  questioning  a  difficult  witness  to  get  the  information  they  need. 

The  next  section  describes  the  threat  assessment  and  questioning  technique 
components  of  the  notional  IEDD  Operator  Course  ITS  instructional  design. 
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3.2.2. 1  Threat  Assessment 

As  previously  discussed,  in  order  for  the  IEDD  operator  to  successfully  dispose  of  the 
explosive  device,  it  is  critical  that  he  or  she  accurately  identifies  the  type  of  device  so 
that  the  correct  Render  Safe  Procedures  (RSPs)  can  be  used.  The  threat  assessment 
process  involves  both  the  close  inspection  of  the  device  itself  and  the  gathering  of 
information  from  third  parties  (e.g.,  civilians,  local  police,  CF  personnel  etc)  through 
questioning  to  determine  the  level  of  threat  posed  to  personnel  by  the  device. 

This  information  can  be  couched  in  terms  of  Situation  Awareness  (SA)  ‘elements’ 
(Endsley,  1995).  Endsley  (1995)  argues  that  these  elements  (i.e. ,  cues  in  the 
environment)  must  be  perceived  and  understood  by  operator  in  order  to  decide  on  and 
initiate  a  successful  course  of  action.  These  SA  elements,  together  with  existing 
knowledge  structures  (i.e.,  mental  models)  about  the  entities  in  the  environment,  form 
the  basis  of  an  operator’s  Situation  Model  of  the  environment. 

In  the  context  of  the  IEDD  Operator  Course,  the  threat  assessment  process  is  the 
means  by  which  an  operator  forms  their  Situation  Model  of  the  suspected  device. 
Specifically,  the  assessment  as  to  what  the  suspected  device  actually  is  will  be  based 
upon  an  ‘argument’  formulated  on  the  basis  of  the  available  evidence  (i.e.,  SA 
elements).  For  example,  evidence  of  a  command  wire  will  support  the  argument  that  the 
suspected  device  is  command-operated,  and  so  on.  Arguments  can  also  be 
conceptualised  in  terms  of  SA  elements  either  supporting  or  objecting  to  the  conclusion. 
For  example: 

•  Conclusion  #1 :  Suspected  device  is  Command-Operated  (or  not) 

o  For:  presence  of  wire  or  transceiver,  suitable  observation  or  vantage  point 
nearby,  presence  of  an  aim  point  (e.g.,  pile  of  rocks  by  roadside),  previous 
insurgent  activity  at  location,  topography  suitable  for  ambush  (e.g.,  culvert, 
bend  in  road),  minimal  civilian  presence. 

o  Against:  Busy  route,  well  patrolled,  civilians  present,  topography  not 
suitable  for  ambush. 

•  Conclusion  #2:  Suspected  device  is  Timer-Operated  (or  not) 

o  For:  predictable  convoy,  presence  of  timer,  intelligence  that  VIP  visits  will 
occur,  convoy  passed  but  no  activation,  festival,  religious  holiday 

o  Against:  High  levels  of  civilian  activity. 

•  Conclusion  #3:  Suspected  device  is  Victim-Operated  (or  not) 
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o  For:  Convoy  passed  but  no  activation,  no  presence  of  command,  no 
Electronic  Warfare  (EW)  detected,  Intel  that  IEED  are  being  targeted, 
primary  device  found, 

o  Against:  busy,  civilian  activity  (inadvertent  activation) 

An  argument  map  is  a  visual  representation  of  the  structure  of  an  argument  in  informal 
logic.  It  includes  the  components  of  an  argument  such  as  a  main  contention  (i.e.,  final 
conclusion),  premises  (i.e.,  a  set  of  one  or  more  declarative  sentences),  and  supporting 
information,  such  as  co-premises  (i.e.,  a  reason  arguing  for  a  premise),  objections  (i.e., 
a  reason  arguing  against  a  premise),  and  rebuttals  (i.e.,  an  objection  to  an  objection). 
Typically  an  argument  map  is  a  ‘box  and  arrow’  diagram  with  boxes  corresponding  to 
premises  and  arrows  corresponding  to  relationships  such  as  evidential  support. 
Argument  mapping  is  often  designed  to  support  deliberation  over  issues,  ideas  and 
arguments  that  are  difficult  or  impossible  to  solve  because  of  incomplete,  contradictory, 
and  changing  requirements  that  are  often  difficult  to  recognize. 

An  argument  map  has  been  created  for  a  hypothetical  IEDD  scenario  (see  Figure  3-1) 
and  represents  the  situation  model  of  the  operator.  The  threat  assessment  activities  that 
took  place  in  order  for  an  operator  to  form  this  situation  model  include  acquisition  of  SA 
elements  through  either  direct  observation  or  the  questioning  of  witnesses.  These  SA 
elements  either  support  or  refute  the  higher-level  premises  or  objections.  The 
advantage  of  using  this  kind  of  approach  is  that  it  enables  the  IEDD  Operator  Course 
ITS  to  make  explicit  the  results  of  the  student’s  threat  assessment  process  (i.e.,  their 
resultant  SA).  In  doing  so,  comparisons  between  an  ‘ideal’  Situation  Model  and  the 
student’s  actual  Situation  Model  can  provide  a  mechanism  for  adapting  the  learning 
experience  or  as  a  focus  for  the  instructor  when  de-briefing  the  student. 
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•*  No  evidence  of  command  w 


CONCLUSION  #1:  No 
evidence  of  COMMAND 
device  components 


No  disturbance  of  road  surface 
No  disturbance  of  surface  by  side  of  road 


*  No  ESM  signatures  detected 


No  evidence  of  wire  in  vicinity 
C  EW  team  report  no  activity 
^  No  evidence  of  cell  phones  being  used  by  civilians 


Rock  pile  by  roadside 
*  indcative  of  aiming  point 


No  evidence  of  receiver  in  vicinity 

•-  Rock  pile  by  side  of  road 
Insurgents  have  used  road  piles  as  an 
aiming  point  in  the  past 


Locals  use  rock  piles  used  to 
1  Civilians  usually  park  by  roadside  *  designate  their  parking  space 


Topography 
1  conducive  to  ambush 


Roadside  culvert  present 


Observation  /  vantage 
*  point  present 


Vantage  point  shows 
*  signs  of  disuse _ 

Observation  /  vantage 
■  point  present 


CF  patrol  found  no 
evidence  of  recent  use 


C  No  recent  attacks  in  the  vicinity 
1  No  recent  IED  attacks  in  this  location  Decrease  in  insurgent 
1  C  activity  recently _ 

Wedding  guests  arrived  at 

1  Civilians  present  C  location  recently _ 


>*  Convoy  passed  without  incident  C  N°  °bser''ed . 


DECISION:  RSPs 
relating  to  VICTIM 
operated  devices  will 
be  deployed 


J 


CONCLUSION  #2:  No 
evidence  of  TIMING 
device  components 


*  No  evidence  of  timing  mechanisms 


Convoy  not  predictable 


No  disturbance  of  road  surface 
No  disturbance  of  surface  by  side  of  road 

I  i  No  evidence  of  timing  device  in  vicinity 
’«  This  route  has  not  been  used  by  CF  for  several  days 

*-  Recent  CF  activity  has  displaced  insurgents 


Recent  attacks  coincided  with  wedding  of 
'  •  rival  tribe  members 


Civilian  wedding  will  be  held 
1  the  next  day 


Wedding  party  is  unlikely  to  be 
*  targeted  by  the  insurgents 


Intel  suspects  some  local 
civilians  support  the 
9  insurgency 


Convoy  passed  without  incident  C  No  insurgent  activity  observed 


CONCLUSION  #3: 
Evidence  of  VICTIM 
device  components 


*  No  evidence  of  command  device  components 


No  disturbance  of  road  surface 
No  disturbance  of  surface  by  side  of  road 


No  evidence  of  timing  device  components 


No  evidence  of  wire  in  vicinity 

•  No  disturbance  of  road  surface 
No  disturbance  of  surface  by  side  of  road 


No  evidence  of  timing  device  in  vicinity 

Intel  suspects  some  local  civilians 
C  support  the  insurgency 

Insurgents  have  not  previously  targeted 
v  106001/  civilians  


'  Civilian  activity  in  the  vicinity 


Insurgents  likely  to  target  IEDD 
operators  despite  risk  of 
1  co-lateral  damage 


Effectiveness  of  insurgency  has  been 
O  reduced  by  CF  counter-IED  measures 

I  IEDD  operators  have  been  recently 
9  targeted  by  secondary  devices 


High  frequency  of  ANA  patrols  during  the  last  week 


’  Recent  ANA  patrols  in  the  vicinity 


LEGEND 


Premise  in  support  of  higher 
•*  level  premise  or  conclusion 

Objection  to  the  higher  level 
^  premise  or  conclusion 


Civilian  dissatisfaction  with 
1  ANA  presence 
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1  known  to  be  ineffective 


Local  civilians  cooperate 
9  against  insurgents _ 

'  9  ANA  personnel  from  a  rival  tribe 

Recent  shura  with  CF 
9  discussed  ANA  corruption 


Situation  Awareness  'element'  acquired  from  the 
environment  (e.g.,  direct  observation)  that  supports 
•  the  higher-level  premise  or  objection 


Situation  Awareness  ’element'  acquired  from  the 
questioning  of  witnesses  that  supports  the 
C  higher-level  premise  or  objection 


Situation  Awareness  'element'  acquired  for  the 
questioning  of  witnesses  that  refutes  the  higher-level 
9  premise  or  objection 


Figure  3-1:  Notional  Situation  Model  for  the  IEDD  Operator  Course  ITS 
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3.2.2.2  Questioning  Technique 

Questioning  Technique  (QT)  is  the  corner  stone  of  IEDD  operations.  It  enables  the 
IEDD  Operator  to  determine  an  appropriate  Render  Safe  Procedure  (RSP)  to  dispose  of 
the  device.  For  effective  threat  assessment,  the  IEDD  Operator’s  best  tools  are  active 
listening  and  questioning.  Within  the  IEDD  operations,  questioning  is  used  to: 

•  Begin  or  continue  discussion; 

•  Pinpoint  and/or  clarify  issues  to  gather  pertinent  information; 

•  Help/direct  the  witness  to  self  explore;  and, 

•  Keep  focus  on  key  issues; 

Good  QT  will  lead  to  more  than  one  answer  and  open  other  avenues  of  questioning. 
Examples  of  good  QT  include  (but  are  not  limited  to):  funnel  questioning,  review 
questions,  and  cross-questioning;  whereas  examples  of  poor  technique  include:  use  of 
jargon  and  leading  questions).  For  example,  a  hypothetical  interaction  between  the 
IEDD  operator  student  and  a  virtual  On  Scene  Commander  (OSC)  within  the  simulated 
scenario  would  be  as  follows: 

•  Student:  How  are  the  civilians  in  this  area? 

•  OSC:  The  local  civilians  are  suspected  to  support  the  insurgency,  although 
nothing  concrete  ...  yet.  However,  it  is  clear  that  they  are  uncooperative  to  CF. 

o  SA  Element:  Local  civilians  may  support  insurgency. 

o  Premise:  Insurgents  will  not  target  local  civilians  directly. 

•  Student:  Ok,  and  what’s  the  deal  with  the  decorations  on  that  house? 

•  OSC:  Their  daughter  is  getting  married  tomorrow.  There  are  lots  of  folks  coming 
into  town  for  this.  It’s  been  the  talk  of  the  town  for  the  longest  time.  The  father  of 
the  bride  is  a  high-profile  figure  here.  We  suspect  he  has  some  links  with  the 
insurgency,  but  again  nothing  concrete. 

o  SA  Element:  Higher  levels  of  civilian  activity  in  the  area  than  normal. 

o  SA  Element:  Father  of  the  bride  may  support  insurgency. 

o  Premise:  Insurgents  are  unlikely  to  target  wedding  party  or  guests. 
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Poor  QT  can  also  lead  to  confirmation  bias  insofar  as  the  questioner  only  selects  data 
that  serves  to  confirm  their  beliefs;  rather  than  seeks  data  to  test  them.  Furthermore, 
despite  having  information  that  contradicts  such  beliefs,  questioners  might  be  very  slow 
to  change  their  minds  (e.g.  Belief  Perseverance). 

Within  the  IEDD  ITS,  the  mission  scenario  can  be  created  in  a  way  that  promotes  the 
phenomena  of  confirmation  bias  and  belief  perseverance  by  providing  easily-accessed 
information  that  at  first  seems  to  support  one  hypothesis,  but  also  providing  more 
difficult  to  access  information  that  refutes  that  hypothesis.  Using  the  situation  model 
described  in  Figure  3-1,  following  an  initial  briefing  that  local  civilians  support  the 
insurgency,  and  that  insurgents  have  not  targeted  ‘friendly’  civilians  in  the  past,  the 
IEDD  operator  might  deduce  that  the  presence  of  civilian  activity  in  the  area  reduces  the 
likelihood  that  the  suspect  device  is  victim-operated.  If  this  initial  hypothesis  remains 
unchallenged  by  conflicting  data,  the  IEDD  operator  is  likely  to  adopt  RSPs  applicable  to 
either  command  or  timer-operated  devices. 

However,  there  are  other  SA  elements  in  the  scenario  which,  if  found,  refute  the 
premise  that  the  presence  of  civilian  activity  in  the  area  precludes  the  possibility  of  a 
victim-operated  device.  In  this  case,  knowledge  that  the  effectives  of  the  insurgency  has 
been  reduced  by  CF  counter-IED  measures,  and  that  IEDD  operators  have  been 
recently  targeted  by  secondary  devices  in  this  area  should  lead  to  the  rebuttal  that 
insurgents  are  likely  to  target  IEDD  operators  despite  the  risk  of  co-lateral  damage  to 
civilians.  This  rebuttal  should  lead  them  to  the  correct  conclusion  that  RSPs  pertaining 
to  a  victim-operated  device  should  be  deployed. 

These  types  of  nuances  within  the  scenario  will  be  exploited  to  fully-exercise  the  range 
of  QT  skills  that  the  student  must  possess  in  order  to  determine  the  correct  RSP  for  the 
device,  and  the  range  of  instructional  interventions  that  can  be  initiated  by  the  IEDD 
ITS. 


3.2.2. 3  Notional  IEDD  Operator  Course  ITS  Hardware  Configuration 

Figure  3-2  describes  a  notional  hardware  setup  for  the  IEDD  ITS.  It  is  anticipated  that 
the  IEDD  ITS  will  be  installed  on  a  stand-alone  computer  with  either  a  monitor 
(specification  to  be  determined),  and  ancillary  mouse  and  keyboard.  Based  on  the 
minimum  hardware  requirements  to  run  the  VBS2  application  and  eye  tracking  software 
simultaneously,  the  hardware  specification  for  the  computer  is  as  follows: 

•  Processor  (CPU):  2.8  GHz  Quad  Core  Processor 

•  Memory  (RAM):  4GB 

•  Hard  Drive:  80GB  free  space 
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•  Video  (Memory):  512MB  Nvidia  Video  Card 

VBS2  is  designed  to  run  on  a  wide  range  of  Commercial-Off-The  Shelf  (COTS) 
computer  systems  readily  available  from  many  vendors.  The  graphics-intensive 
requirements  however  dictate  special  consideration  to  certain  components  such  as 
the  video  card  models  configured  in  the  system.  Further  information  can  be  found  at 
Bohemia  Interactive’s  VBS2  website  http://virtualbattlespace.vbs2.com 


PSYCHOPHYSIOLOGICAL 

TRACKING 


EYE  TRACKING 


% 

BEHAVIOUR 

TRACKING 


Figure  3-2:  Notional  hardware  setup  for  the  IEDD  ITS 


The  IEDD  ITS  will  utilise  eye-tracking  and  psychophysiological  tracking  devices  as 
inputs  to  the  adaptation  mechanism.  The  intention  is  that  both  these  tracking 
technologies  will  be  as  unobtrusive  as  possible;  in  terms  of  their  ease  of  set-up  and 
configuration  and  their  robustness  to  frequent  use  by  course  students.  The  technical 
specifications  of  these  technologies  remain  to  be  determined;  however  it  is  anticipated 
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that  both  types  of  technologies  will  provide  real-time  data  (i.e. ,  gaze  location  and 
duration,  and  a  numerical  value  of  HRV2)  to  the  ITS. 


3.2.2.4  Notional  IEDD  Operator  Course  ITS  User  Experience 

A  notional  IEDD  Operator  Course  ITS  user  experience  is  presented  in  Figure  3-3  and 
illustrates  the  key  components  of  the  Graphical  User  Interface  (GUI)  in  terms  of  both  the 
presentation  of  training  content  and  the  means  by  which  the  student  interacts  with  the 
IEDD  ITS.  The  GUI  is  divided  into  the  following  components,  which  are  described  in 
relation  to  the  notional  GUI  presented  in  Figure  3-3: 

•  Simulated  scenario  environment.  Presented  in  the  top  left  hand  window  of  the 
GUI,  the  simulated  environment  is  the  primary  means  by  which  the  student 
interacts  with  the  scenario  environment.  This  frame  will  display  a  ‘first-person 
shooter’  (FPS)  view  of  the  scenario  environment,  which  will  include  terrain, 
vehicles,  lED-related  cues  (e.g.,  wires,  road  patches,  rock  piles,  and  so  on),  as 
well  as  the  scenario  Non-Player  Characters  (NPCs).  Students  can  navigate 
around  the  environment  using  the  cursor  keys  on  the  keyboard.  As  with  most 
FPSs,  VBS2  takes  over  the  mouse,  and  mouse  motion  moves  the  gaze  around. 
The  student  cannot  click  on  objects  with  the  mouse.  Instead,  the  selection 
mechanism  in  VBS2  is  to  approach  the  object  and  hit  the  ‘enter’  key  or  ‘space 
bar’  on  the  keyboard.  For  fine  selection,  the  student  alters  their  gaze  with  the 
mouse  until  the  crosshairs  in  the  middle  of  the  screen  is  over  the  object,  and  then 
hits  the  ‘space  bar’  or  ‘enter’  key  to  make  a  selection.  The  NPCs  are  identified 
using  a  yellow  arrow  and  a  label.  If  a  student  wishes  to  question  a  witness,  they 
need  to  walk  up  to  the  NPC  and  initiate  the  dialogue  using  the  ‘enter’  key  or 
‘space  bar’  on  the  keyboard.  When  the  dialogue  is  initiated,  the  arrow  above  the 
NPC  will  turn  red.  After  the  dialogue  is  completed,  the  arrow  will  return  to  a 
yellow  colour.  Finally,  the  student  will  be  able  to  click  on  relevant  cues  observed 
within  the  environment  (using  the  crosshairs  and  the  keyboard  entry  as  described 
above)  to  activate  them  within  the  representation  of  their  threat  assessment  (see 
below). 

•  NPC  Dialogue.  The  dialogue  with  the  NPCs  is  presented  within  the  top  right 
window  of  the  GUI  and  is  the  primary  means  by  which  the  student  questions  the 
scenario  witnesses.  This  window  displays  a  head  and  shoulders  close-up  of  the 
NPC’s  avatar  to  enable  any  facial  expressions  (if  provided  by  the  supporting 


2 

In  order  for  the  IEDD  ITS  to  adapt  the  learning  experience  of  the  student  in  response  to  changes  in  their  HRV  data,  we  first  need  to 
determine  the  appropriate  response  ‘thresholds’  for  these  data.  These  thresholds  will  be  unique  to  each  student  and  are  calculated  from 
baseline  data  collected  from  the  student  under  ‘normal’  resting-state  (e.g.,  relaxed,  unstimulated)  conditions.  In  order  to  determine  the 
threshold  for  what  can  be  considered  an  ‘affective  response’,  the  student  needs  to  be  presented  with  stimulus  material  appropriate  to  the 
scenario  (e.g.,  photos  of  IED  components  or  IED  scenes,  videos  of  IEDD  operators  conducting  disposal  and  so  on)  during  which  HRV  data 
are  collected.  The  amount  of  change  in  the  HRV  data  required  to  trigger  an  adaptation  by  the  ITS  can  be  calculated.  In  conclusion,  the  IEDD 
ITS  should  include  a  psychophysiological  device  calibration  exercise,  which  could  be  performed  under  the  auspices  of  a  short  introductory 
video  or  presentation  outlining  the  project  goals  and  the  importance  of  threat  assessment  to  successful  IED  disposal. 
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software)  to  be  observed  by  the  student.  Each  round  of  dialogue  is  presented  in 
this  window;  however  no  history  of  the  dialogue  is  recorded.  This  is  to  ensure 
that  the  student  memorises  the  content  of  the  dialogue;  as  in  real-life.  The 
dialogue  will  be  turn-based  insofar  as  the  student  will  ask  a  question  and  the 
NPC  will  provide  an  answer  to  that  question.  Finally,  the  student  will  be  able  to 
select  from  a  list  of  questions  which  one  they  would  like  to  ask  the  NPC.  The 
selection  of  questions  is  either  keyboard-based  (a  numerical  identifier  is  allocated 
to  each  option),  or  mouse-based  (point  and  click  on  the  option).  The  list 
presented  in  Figure  3-3  is  for  illustrative  purposes  only.  Care  will  be  taken  during 
the  development  of  this  functionality  to  ensure  that  the  list  of  questioning  options 
does  not  constrain  students’  choices  and  pre-dispose  them  to  adopting  a 
particular  questioning  style.  One  option  would  be  to  present  students  the 
categories  of  questions  (e.g.,  “questions  about”)  and  then  provide  a  list  of  options 
for  asking  the  same  question  but  in  different  ways.  This  should  allow  students  to 
exhibit  their  preference  for  a  particular  questioning  style. 


I  COMMAND 


DECISION:  RSPs 
relating  to  [SELECT] 
operated  devices  will 
be  deployed 


You:  How  are  the  civilians  in  this  area? 


OSC:  The  local  civilians  are  suspected 
to  support  the  insurgency,  although 
nothing  concrete  ...  yet.  However,  it  is 
clear  that  they  are  uncooperative  to  CF. 


You:  Ok,  and  what's  the  deal  with  the 
decorations  on  that  house? 


OSC:  Their  daughter  is  getting  married 
tomorrow.  There  are  lots  of  folks 
coming  into  town  for  this.  It's  been  the 
talk  of  the  town  for  the  longest  time. 
The  father  of  the  bride  is  a  high-profile 
figure  here.  We  suspect  he  has  some 
links  with  the  insurgency,  but  again 
nothing  concrete. 


1.  Thank  you  Sir,  that  will  be  all.  2.  Weddings  have  been  attacked  before,  is  this  one  at  risk  too?  3.  Who  is  she  marrying? 

4.  Tell  me  more  about  the  father  of  the  bride?  5.  Are  the  wedding  guests  all  from  the  same  tribe? 


INSTRUCTOR 


COMMAND 


You  selected  an  open-ended 
question.  You  should  be  more 
specific. 


TIMING 


Also,  you  seemed  to  have  missed 
the  significance  of  the  local 
civilians  supporting  the 
insurgency.  What's  the 
importance  of  that? 


VICTIM 


Figure  3-3:  Notional  IEDD  Operator  Course  ITS  User  Experience 
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•  Threat  Assessment  (Situation  Model  Argument  Map).  As  discussed  in  the 
previous  section,  we  have  proposed  that  an  argument  map  is  used  to  externalise 
the  student’s  threat  assessment  and  decision  making  processes  when  working 
through  the  IEDD  scenario.  The  Situation  Model  Argument  Map  (SMAP)  is 
presented  in  the  bottom  right  window  of  the  GUI,  and  provides  a  real-time 
depiction  of  the  SA  elements  observed  directly  from  the  environment  (as 
determined  by  the  student’s  selections  in  the  scene3)  and  the  SA  elements 
acquired  from  the  questioning  of  witnesses  (as  determined  by  the  content  of  the 
dialogue).  SA  elements  acquired  through  direct  observation  will  be  automatically 
populated  into  the  SMAP  by  the  ITS.  The  SMAP  will  offer  some  degree  of 
interactivity  in  terms  of  the  student  will  be  required  to  manually  populate  the  SA 
elements  acquired  from  the  questioning  of  witnesses  (e.g.,  click  and  drag  the  SA 
element  to  a  premise  or  objection  in  the  SMAP),  and  the  construction  of  the 
higher  order  premises  (or  objections)  derived  from  the  SA  elements.  By  forcing 
the  student  to  make  their  deductions  explicit,  it  is  hoped  that  the  ITS  will  be  able 
to  monitor  the  on-set  of  cognitive  biases  (e.g.,  confirmation  bias).  Finally,  the 
student  will  have  the  ability  to  flag  which  type  of  device  they  currently  believe  the 
suspect  device  to  be  (i.e.,  command,  timer  or  victim-operated).  The  student  can 
make  a  final  assessment  of  the  device  at  any  time  they  choose  during  the 
scenario.  This  effectively  ends  the  scenario. 

•  Virtual  Instructor.  The  interaction  with  the  Virtual  Instructor  is  presented  within  the 
bottom  left  window  of  the  GUI  and  is  the  primary  means  by  which  the  ITS 
explicitly  coaches  the  student  regarding  their  QT  and  threat  assessment  skills.  As 
discussed  before,  the  advantage  of  using  the  SMAP  is  that  it  enables  the  ITS  to 
make  comparisons  between  an  ‘ideal’  Situation  Model  and  the  student’s  actual 
Situation  Model.  This  knowledge  provides  a  mechanism  for  adapting  the  learning 
experience.  In  addition,  the  Virtual  Instructor  will  also  have  access  to  the  eye¬ 
tracking  data,  mouse  clicks  on  the  scenario  simulation  window,  and  measures  of 
the  student’s  affective  response  from  the  psychophysiological  tracking.  This 
information  could  be  used  by  the  Virtual  Instructor  to  ‘hint’  to  the  student  that  they 
might  have  missed  key  SA  elements.  For  example,  although  the  eye-tracking 
data  might  indicate  that  they  were  looking  at  an  important  visual  cue  (e.g.,  pile  of 
rocks),  a  lack  of  a  selection  on  the  rock  pile  would  suggest  that  they  did  not  think 
it  was  important.  Alternatively,  when  receiving  an  answer  from  a  NPC  which 
contains  an  important  piece  of  information,  a  lack  of  affective  response  from  the 
psychophysiological  tracking  would  suggest  that  they  did  not  think  it  was 
important  (together  with  no  attempt  to  update  the  SMAP  with  this  SA  element).  In 
both  cases,  the  Virtual  Instructor  would  intervene,  and  hint  that  the  student  might 
want  to  revisit  some  of  these  cues. 


The  project  team  will  need  to  identify  how  to  handle  ‘false  alarms’;  in  other  words,  elements  in  the  scene  that  were  selected  by  the  student, 
but  are  not  considered  to  be  relevant  to  the  threat  assessment.  These  data  could  provide  insight  into  the  deficiencies  of  a  student’s  threat 
assessment  technique  (e.g.,  confirmation  bias). 
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3.3  Recommended  Architecture  for  the  IEDD  Operator  Course  ITS 

Training  systems  and  technologies  are  evolving  to  the  point  where  the  implementation 
of  adaptive  learning  systems  is  possible.  The  results  of  the  literature  review  seem  to 
indicate  that  the  increased  training  efficiency  of  adaptive  systems  more  than  offsets  the 
increased  cost  and  complexity  of  these  systems.  However,  standards  such  as  SCORM 
are  still  playing  catch  up  to  handle  advanced  training  technologies  such  as  simulation- 
based  training  and  virtual  world  collaborative  environments.  DND  enterprise  learning 
systems  such  as  DNDLearn  and  AFIILE  do  not  currently  support  these  evolving  training 
delivery  modalities.  For  this  reason,  it  is  strongly  recommended  that  initial  adaptive 
learning  technology  development  should  not  be  tightly  coupled  to  the  DND  enterprise 
training  systems  as  currently  implemented. 

Adaptive  learning  technology  developments  must  be  designed  for  eventual  inclusion  in 
systems  such  as  DNDLearn  and  AFIILE.  However,  integration  with  these  systems  as 
they  are  implemented  now  will  preclude,  or  limit,  the  use  of  many  advanced  training 
delivery  modalities  such  as: 

•  Simulation  environments  that  have  specific  hardware  requirements  for  the 
training  platform,  such  as  graphics  capabilities  or  human-computer  interface 
devices; 

•  Simulation  environments  that  have  extensive  data  requirements,  such  as 
extensive,  photo-realistic  virtual  terrains; 

•  Simulation  environments  that  operate  in  networked,  multiplayer  modes; 

•  Tracking  of  student  performance  data  in  forms  other  than  SCORM;  and, 

•  Design  and  delivery  of  suggested  notional  learning  interface  that  supports 
intelligent  tutoring  and  adaptive  capabilities. 

Furthermore,  it  will  be  important  that  there  are  good  channels  of  communication 
between  the  adaptive  learning  work  and  DND’s  learning  system  developers  to  inform 
the  development  DND  enterprise’s  learning  capabilities. 

A  three-stage  process  is  recommended.  First,  an  adaptive  learning  framework  should 
be  assembled.  Second,  learning  content  components  of  an  electronic  IEDD  course 
should  be  developed.  While  future  inclusion  in  a  learning  management  system  must  be 
considered,  these  learning  content  components  should  not  be  limited  to  technologies 
currently  compatible  with  DND’s  learning  management  systems.  Finally,  the  IEDD 
learning  content  components  should  be  integrated  with  the  adaptive  learning  framework 
to  produce  a  training  delivery  platform  capable  of  performing  experimental  delivery  of 
adaptive  training  using  the  electronic  IEDD  courseware  components. 
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The  adaptive  learning  framework  should  be  a  generic  experimental  framework  that 
enables  experimental  training  delivery  with  the  potential  to  vary: 

•  The  training  content  (course)  delivered; 

•  The  training  content  delivery  type  (text,  imagery,  multimedia,  virtual  simulation); 

•  The  training  content  delivery  style  (individual,  collaborative,  instructor-led); 

•  The  content  adaptation  mechanism  (sequencing,  pacing,  complexity,  type, 
learning  style,  etc);  and, 

•  The  component  technologies  used  in  the  adaptation  framework. 

A  notional  architecture  for  the  adaptive  learning  framework  is  given  in  Figure  3-4.  The 
adaptive  learning  framework  should  incorporate  learning  content  and  student 
information  management  capabilities  for  experiment  execution  and  data  collection 
purposes.  Architecture  and  design  of  the  requirements  of  the  adaptive  framework 
should  take  into  consider  future  integration  potential  with  existing  DND  Learning 
Content  Management  and  Learning  Management  Systems  (LCMS  and  LMS). 


Figure  3-4:  Conceptual  architecture  for  an  adaptive  learning  system  (from  Banbury  et  al., 

2009) 
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The  functionality  required  for  the  adaptive  framework  will  be  similar  to  the  functionality 
within  current  DND  LMSs.  The  adaptive  learning  framework  should  not  be  required  to 
integrate  with  current  DND  LMS  implementations,  as  this  could  potentially  preclude  the 
use  of  certain  learning  content  types  and  delivery  mechanisms.  Rather,  the  format  and 
standards  involved  with  passing  data  between  the  systems  will  be  considered  during  the 
design  phase  of  the  ITS  project. 

The  components  of  the  recommended  adaptive  learning  framework  are  outlined  below 
and  detailed  in  the  following  sections: 

•  Training  Delivery  Module(s).  The  purpose  of  the  training  delivery  module  is  to 
present  the  training  content  to  the  student.  This  is  the  component  of  the  system 
with  which  the  student  interacts.  The  training  delivery  module  must  be  capable  of 
presenting,  at  minimum:  text-based  content,  multimedia  content,  individual  virtual 
simulation  content,  and  collaborative  virtual  simulation  content.  The  presentation 
of  training  content  is  controlled  by  the  adaptation  module.  The  training  delivery 
module  must  report  outcomes  and  student  interactions  as  required  to  the 
evaluation  module.  A  training  delivery  module  instance  will  be  required  for  each 
student  participating  concurrently  in  a  training  experiment; 

•  Evaluation  Module.  The  purpose  of  the  evaluation  module  is  to  update  the 
student  model  based  on  measurements  of  the  student.  The  evaluation  module 
must  be  capable  of  direct  measurements  of  the  student  using  physiological 
measuring  devices,  as  well  as  indirect  monitoring  of  student  performance  from 
user  interactions  reported  by  the  training  delivery  module,  such  as  key  presses, 
timings,  quiz  results,  and  so  on; 

•  Adaptation  Module.  The  purpose  of  the  adaptation  module  is  to  adapt  the 
delivery  of  training  based  on  the  evolution  of  the  student  model.  The  adaptation 
module  should  examine  the  student  model  to  adapt  the  delivered  learning  style  to 
the  evolving  understanding  of  the  student’s  learning  style.  The  adaptation  module 
must  examine  the  difference  between  the  student’s  grasp  of  the  subject  matter, 
the  required  understanding  of  the  subject  matter,  and  the  expert  representation  in 
order  to  adapt  any  required  aspects  of  the  training  delivery  to  improve  the 
efficiency  of  the  training  delivery; 

•  Expert  Model.  The  purpose  of  the  expert  model  is  to  represent  the  knowledge, 
skills  and/or  behaviours  that  embody  the  desired  end  state  of  the  student.  The 
expert  model  must  encapsulate,  at  minimum,  the  required  proficiency  of  the 
student,  and  potentially  the  desired  expert  proficiency  beyond  that.  The  expert 
model  is  required,  by  the  evaluation  and  adaptation  modules,  to  evaluate  the 
difference  between  the  student’s  current  state  and  the  desired  learning  outcome. 
The  expert  model  will  not  change  during  training  delivery, 
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•  Student  Model.  The  purpose  of  the  student  model  is  to  represent  the  current 
knowledge,  skills  and/or  behaviours  that  embody  the  student.  The  student  model 
will  be  updated  during  the  delivery  of  training  based  on  measurements  of  the 
student  performed  by  the  evaluation  model.  The  student  model  may  also  require 
historical  information  on  the  student,  such  as  learning  history.  The  student  model 
is  required,  by  the  evaluation  and  assessment  modules,  to  evaluate  the 
difference  between  the  student’s  current  state  and  the  desired  learning  outcome; 

•  Experiment  Control  Module.  The  purpose  of  the  experiment  control  module  is  to 
control  the  execution  of  the  adaptive  learning  framework  in  a  controlled  fashion 
that  collects  the  required  data.  The  experiment  control  module  will,  at  minimum, 
be  required  to  provide  initialization  data  (such  as  student  history)  as  required  by 
the  various  modules,  and  to  collect  experimental  data  as  required  from  the 
various  modules;  and 

•  Instructor  Module(s).  The  purpose  of  the  instructor  module  is  to  allow  the 
instructor  to  interact  with  the  student(s)  when  requested  by  the  student  or 
instructor,  or  required  by  the  training  content.  The  instructor  should  be  capable  of 
participating  collaboratively  with  the  student(s)  in  the  training  content.  The 
instructor  should  be  capable  of  collaborating  with  the  student  outside  of  the 
training  content  in  order  to  be  able  to  coach  or  assist  the  student  in  an 
unstructured  fashion.  Depending  on  the  course  content,  the  instructor  may  be 
required  to  enter  assessment  information  based  on  their  assessment  of  the 
student’s  skill  or  performance.  An  instructor  module  will  be  required  for  each 
instructor  involved  in  the  experiment,  as  required  by  the  training  content. 

•  Blackboard.  A  blackboard  system  is  an  artificial  intelligence  application  based  on 
the  blackboard  architectural  model,  where  a  common  knowledge  base,  the 
"blackboard",  is  iteratively  updated  by  a  diverse  group  of  specialist  knowledge 
sources,  starting  with  a  problem  specification  and  ending  with  a  solution.  Each 
knowledge  source  updates  the  blackboard  with  a  partial  solution  when  its  internal 
constraints  match  the  blackboard  state.  In  this  way,  the  specialists  work  together 
to  solve  the  problem.  The  blackboard  model  was  originally  designed  as  a  way  to 
handle  complex,  ill-defined  problems,  where  the  solution  is  the  sum  of  its  parts. 
Both  the  Experiment  Control  and  the  Instructor  Modules  could  make  use  of 
Blackboards. 

It  should  be  noted  that  a  limited  adaptive  learning  framework  can  be  constructed  within 
a  SCORM  compliant  LMS  component  at  this  time.  In  such  a  system,  the  components 
listed  above  are  populated  in  the  following  fashion: 

•  The  training  delivery  module  is  a  standards  compliant  browser,  and  the  training 
content  is  one  or  more  SCO’s  limited  to  the  multimedia  content  that  can  be 
displayed  in  a  browser  without  any  external  software  dependencies; 
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•  The  evaluation  module  is  embedded  in  the  SCOs  and  is  reflected  by  the  success 
or  failure  that  the  SCOs  report  to  the  LMS; 

•  The  adaptation  module  is  limited  to  the  branching  decisions  that  can  be 
represented  in  the  SCO  sequencing  information  (for  instance,  remedial  content 
can  be  presented  if  the  student  scores  lower  than  a  given  standard  on  an 
intermediate  evaluation); 

•  The  expert  model  is  limited  to  the  lesson’s  success  criteria  (score)  in  the  SCO 
sequencing  information; 

•  The  student  model  is  limited  to  the  lesson  outcomes  reported  by  the  involved 
SCOs; 

•  The  experiment  control  module  is  the  containing  LMS;  and, 

•  Instructor  module  components  for  unstructured  interaction  with  the  student  (such 
as  chat,  email,  video  conferencing)  are  sometimes  provided  by  the  LMS. 

The  adaptive  learning  framework  is  detailed  further  in  the  following  sections. 


3.3.1  Training  Delivery  Module 

The  purpose  of  the  training  delivery  module  is  to  present  the  training  content  to  the 
student,  and  to  manage  the  operator  machine  interface  (OMI).  The  training  delivery 
module  should  be  based  on  a  standard,  3D  graphics  capable  PC  with  keyboard,  mouse, 
headset  with  microphone,  web  camera  and  a  single  monitor.  Other  input  devices  such 
as  joysticks  and  game  controllers  should  be  considered. 

The  delivery  of  training  content  is  controlled  by  the  adaptation  module.  The  training 
delivery  module  must  present  the  required  content  and  report  required  student 
interactions  to  the  evaluation  module.  As  the  training  content  selection  is  controlled  by 
the  adaptation  module,  the  software  presenting  different  content  types  can  be  treated 
independently,  and  may  potentially  be  presented  simultaneously  (such  as  text  to  image 
reference  material  alongside  virtual  simulation-based  practice).  The  required  content 
types  that  must  be  supported  during  an  experiment  are  determined  by  the  course  that  is 
presented  -  the  training  delivery  module  must  be  capable  of  supporting  the  content 
types  used  in  the  electronic  courseware.  The  following  list  defines  content  types  that 
should  be  supported,  and  the  recommended  software  applications: 

•  Text,  Images  and  Multimedia.  These  content  types  should  be  delivered  in  a 
standards-compliant  web-browser  using  standards-based  browser  technologies 
such  as  XHTML,  videos,  Adobe  Flash  and  JavaScript.  This  content  could  be 
static  or  interactive,  and  might  contain  elements  that  explicitly  evaluate  the 
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student  through  structured  quizzes.  If  possible,  this  category  of  content  handling 
should  be  performed  in  a  fashion  that  does  not  preclude  SCORM  compliance.  At 
the  time  of  this  report,  the  recommended  software  for  delivery  of  this  content-type 
is  Bohemia  Interactive’s  VBS2.  This  software  was  selected  in  part  due  to  its 
prevalence  and  availability  within  DND. 

•  Immersive  Virtual  Simulation.  This  content-type  would  include  realistic,  3D 
simulation  of  a  3D  space,  such  as  the  terrain  and  features  surrounding  the  area 
of  a  reported  IED  location.  The  student  might  interact  with  this  content-type 
individually  or  with  other  students  and/or  instructors.  This  content-type  might  be 
used  for  unstructured  practice,  structured  learning  or  instructor-led  learning.  The 
student  might  be  evaluated  explicitly  or  assessed  implicitly  through  their 
interactions  with  this  content-type.  The  recommended  software  for  delivery  of  this 
content-type  is  Bohemia  Interactive’s  VBS2, 

•  Non-lmmersive  Virtual  Simulation.  This  content-type  would  include  the  detailed, 
realistic  simulation  of  an  IED  in  isolation  or  in  the  context  of  the  immediate 
environment  at  a  nuts,  bolts  and  wires  level.  The  student  might  interact  with  this 
content-type  individually  or  with  other  students  and/or  instructors.  This  content- 
type  might  be  used  for  unstructured  practice,  structured  learning  or  instructor-led 
learning.  The  student  might  be  evaluated  explicitly  or  assessed  implicitly  through 
their  interactions  with  this  content  type.  If  possible,  existing  learning  objects  or 
other  existing  content  will  be  identified  and  be  re-purposed  for  this  simulation. 
More  specifically,  DND  has  procured  nGrain  IED  and  bomb  models.  With  some 
effort,  these  models  may  be  able  to  be  re-purposed.  Furthermore,  some  types  of 
lEDs  may  have  a  nGrain  Virtual  Task  Trainer  associated  with  them.  This  virtual 
task  trainer  often  supports  the  functionality  described  above  and  should  be 
integrated  if  possible  rather  than  creating  a  duplicate  function.  Benefits  and 
savings  of  re-purposing  and  integration  efforts  will  be  examined  on  a  case-by- 
case  basis.  The  recommended  software  for  the  delivery  of  this  content  type  is 
nGrain’s  models.  Should  models  not  be  suitable  or  if  desired  models  do  not  exist, 
development  of  new  models  may  be  developed  using  nGrain  technology  or  other 
readily  available  modeling  technology. 

The  training  delivery  module  must  report  outcomes  and  student  interactions  as  required 
to  the  evaluation  module.  Where  possible,  this  reporting  process  should  align  with  the 
SCORM  standard.  An  assessment  of  current  and  future  SCORM  standards  should  be 
considered. 

Other  supporting  frameworks  for  student  collaboration  and  student/instructor  interaction 
should  be  considered  when  necessary  by  training  course  requirements.  As  these 
components  do  not  integrate  with  the  others  of  the  adaptive  learning  framework,  the 
selection  of  technologies  is  not  critical.  However,  the  following  applications  are 
recommended: 
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•  Skype  for  video  telephony; 

•  MediaWiki  for  unstructured  text  based  collaboration  and  knowledge  sharing;  and, 

•  Access  to  standard  email  and  text  chat,  with  provisions  for  students  and 
instructors  who  do  not  have  or  do  not  wish  to  use  personal  or  business  accounts. 

The  technologies  outlined  above  have  been  selected  to  ensure  focus  on  the  adaptive 
engine  and  reduce  concerns  of  licensing  issues,  and  so  on.  It  should  be  noted,  however 
that  these  functionalities  will  become  available  through  the  DND  enterprise  integrated 
information  learning  environment.  Integration  of  these  future  functionalities  should  be 
considered  during  design.  A  training  delivery  module  instance  will  be  required  for  every 
student  participating  in  the  training  session. 


3. 3. 1.1  High  Level  Requirements 

For  the  IEDD  course,  the  high  level  requirements  for  the  Training  Delivery  Module  are: 

•  Present  3D  scene  to  student: 

o  Allow  navigation  of  scene; 
o  Allow  visual  inspection  of  scene; 

o  Show  Non-Player  Character  (NPC)  locations  and  actions; 
o  Interaction  with  scene  objects;  and, 

o  Show  active  elements  of  scene,  such  as  people  moving,  and  potentially 
device  detonations. 

•  Allow  questioning  of  NPCs: 

o  Selection  of  NPC  to  engage  in  questioning; 
o  Select  questions;  and, 
o  Receive  answers. 

•  Potentially  allow  ordering  of  NPCs: 

o  Select  NPC; 
o  Select  order;  and, 
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o  Receive  response. 

•  View  current  state  of  situation  as  described  by  the  SMAP; 

•  Allow  student  to  make  deductions  (correct  or  incorrect); 

•  Allow  student  to  record  observations  (significant  or  insignificant);  and, 

•  Present  intelligent  tutor  coaching  to  the  student  in  text,  audio  or  multi-media. 

3. 3. 1.2  Relationship  with  Other  Modules 

The  training  delivery  module  accepts  input  from  the  adaption  module  and  produces 
output  for  the  evaluation  module.  Key  inputs  for  this  module  will  be: 

•  Site  data; 

•  Scenario  event  timeline; 

•  Question  trees; 

•  Deduction  tree/structure;  and, 

•  Multimedia  coaching  packages. 

Key  outputs  from  this  module  will  be: 

•  Student  location  and  orientation; 

•  Student  NPC  selections; 

•  Student  question  selections; 

•  Student  deductions;  and, 

•  Student  observations. 

3. 3. 1.3  Recommended  Implementation 

In  the  context  of  the  IEDD  course,  the  training  module  enables  the  student’s  interaction 
with  the  IED  site  and  the  Non-Player  Characters  (NPC’s)  in  the  area.  The  student  will 
navigate  a  3D  scene  to  observe  and  manipulate  aspects  of  the  site,  and  to  select  NPC’s 
for  questioning.  Questioning  will  take  place  using  a  text  based  interaction,  with  the 
student  selecting  from  predetermined  questions  representing  a  range  of  questioning 
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styles  and  a  variety  of  topics  wider  than  the  IEDD  focus.  An  element  of  the  display  will 
allow  the  intelligent  tutor  to  provide  coaching  to  the  student. 

The  training  delivery  module  is  responsible  for  implementing  the  user  experience 
outlined  in  Figure  3-4.  Bohemia  Interactive  VBS2  is  recommended  to  implement  the  3D 
window  for  scene  elements,  navigation  and  selecting  NPC’s  for  interaction  (in  the  upper 
left  in  the  concept  GUI).  Alelo  Virtual  Role  Players  are  recommended  for  further 
investigation  to  implement  the  NPC  dialog  functionality  surrounding  the  3D  window. 
COTS  multimedia  display  technology  such  as  Adobe  Flash  is  recommended  for  the 
presentation  of  the  SMAP  and  the  intelligent  tutoring  coaching  material. 

Bohemia  Interactive’s  VBS2  provide  a  large  library  of  equipment  and  scene  items,  and 
provide  support  for  training  goals  relevant  to  the  IEDD  course,  including: 

•  Assessing  the  scene; 

•  Enforcing  scene  safety; 

•  Evacuating  personnel; 

•  Provide  Command  and  Control  (C2); 

•  Establish  cordon; 

•  Establish  ICP;  and, 

•  Forensic  evidence  gathering. 

VBS2  includes  an  extensive  library  of  equipment  and  models,  as  well  as  content 
creation,  scenario  generation  and  after  action  review  tools.  Development  will  be 
required  to  interface  the  simulation  environment  into  the  adaptive  learning  framework, 
such  that  the  evaluation  module  can  make  use  of  information  from  the  simulation 
environment,  and  so  that  the  adaptation  module  can  control  the  execution  of  the 
simulation  environment  (see  Figure  3-5  for  an  example). 
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Figure  3-5:  A  screenshot  of  existing  immersive,  virtual  IED  training  material  developed  in 

VBS2 


Alelo’s  Virtual  Role  Players  for  VBS2  allows  instructors  to  populate  VBS2  worlds  with 
intelligent  virtual  role  players,  which  communicate  using  spoken  foreign  language  and 
exhibit  culturally  appropriate  behaviour.  VRP-VBS2  makes  it  possible  to  create  highly 
realistic  training  environments  in  which  students  can  practice  their  intercultural 
competency  and  foreign  language  skills,  not  just  their  kinetic  warfighting  skills.  The 
avatar  interaction  approach  and  technology  underlying  Alelo’s  products  should  be 
investigated  further  to  support  the  questioning  technique  element  of  the  Training 
Delivery  Module  (see  Figure  3-6  for  an  example). 
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Figure  3-6:  A  screenshot  of  existing  Alelo  cultural  training  within  VBS2  (top  left  actions, 

top  right  output) 


3.3.2  Evaluation  Module 

The  purpose  of  the  evaluation  module  is  to  update  the  student  model  based  on 
measurements  of  the  student.  The  evaluation  module  must  be  capable  of  direct 
measurements  of  the  student  using  physiological  measuring  devices,  as  well  as  indirect 
monitoring  of  student  performance  from  user  interactions  reported  by  the  training 
delivery  module.  Due  to  the  experimental  nature  of  the  recommended  integration  of 
physiological  measurements  with  virtual  simulation,  the  evaluation  module  will  involve  a 
significant  proportion  of  custom  development. 

As  measurement  technologies  and  evaluation  strategies  will  be  a  focus  of  the  adaptive 
learning  experimental  plan,  the  evaluation  module  should  be  designed  in  a  flexible, 
componentized  fashion.  Components  that  should  be  incorporated  are: 

•  Eye  Tracking.  Eye  tracking  affords  insight  into  the  student’s  locus  of  attention. 
This  information  can  be  used  by  both  the  instructor  and  the  system.  The  visual 
significance  attributed  by  the  student  to  features  in  a  simulated  scene  can  be 
compared  to  significances  obtained  from  the  training  objectives  or  from  tracking 
data  recorded  from  subject  matter  experts. 
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•  Heart  Rate  Variability  (HRV).  This  technology  can  be  used  as  an  indicator  of  the 
students’  level  of  excitement  or  relaxation. 

•  Student  Interaction  Measures.  Information  from  the  training  delivery  can  be 
compared  to  knowledge  of  the  training  content  to  obtain  measures  of  student 
proficiency,  pace  and  behaviour.  For  example,  a  student  who  ‘s  performance 
closely  follows  the  expert  path  will  likely  accomplish  tasks  in  the  scenario  in  a 
timely  fashion,  use  fewer  on-line  references  if  available,  and  perform  tasks 
correctly  in  fewer  attempts.  Where  possible,  this  information  should  be  handled  in 
a  fashion  that  does  not  preclude  SCORM  compliance.  However,  the 
implementation  limitations  of  SCORM  around  virtual  simulation  dictate  that  full 
SCORM  compliance  will  not  be  possible. 

The  evaluation  module  must  process  the  student’s  measurements  into  an  assessment 
of  the  student  state  (workload,  stress,  proficiency,  strategy  employed,  etc.)  and  use  this 
information  to  update  the  student  model.  Due  to  the  experimental  nature  of  this  process, 
it  is  likely  that  development  will  be  required  for  this  capability.  Section  3.1  of  this  report 
reviews  the  results  of  parallel  activities  undertaken  to  review  and  identify  adaptation 
mechanisms  for  adaptive  learning  and  intelligent  tutoring  technologies  (see  Kramer, 
Tryan  and  Banbury;  2010).  This  section  provides  recommendations  for  the  eye-tracking, 
psychophysiological  indices  of  workload  and  stress,  learning  styles  and  performance 
tracking  technologies. 

The  content  delivery  platforms  detailed  above  (traditional  e-learning,  immersive 
simulation  and  non-immersive  simulation)  all  contain  provisions  for  proficiency 
assessment  through  measurements  of  performance  objectives  -  scores  on  exams, 
completion  of  performance  objectives  and  so  on. 

These  standard  methods  of  e-learning  proficiency  assessment  can  be  included  in  the 
training  content  objects  where  suggested  by  the  training  needs  analysis  and 
pedagogical  design.  These  methods  provide  feedback  to  the  student  and  the  adaptation 
module  on  how  the  student  performed. 

Further,  the  instructor  module  enables  the  instructor  to  provide  feedback  on  the  student. 
Beyond  proficiency  measurements,  an  instructor  can  normally  provide  insight  into  the 
reasons  for  the  success  or  failure,  and  suggest  corrected  courses  of  action.  This 
information  is  of  high  value  to  the  student.  Also,  if  this  information  can  be  captured  in  a 
fashion  amenable  to  machine  processing,  it  should  be  considered  by  the  adaptation 
module  as  an  adaptation  input.  This  information  can  characterise  why  the  student 
performed  the  way  they  did. 


28  May  201 0  -  54  -  5045-1  Version  02 

©  Her  Majesty  the  Queen  as  represented  by  the  Minister  of  National  Defence,  2010 
©  Sa  majeste  la  reine,  representee  par  le  ministre  de  la  Defense  nationale,  2010 


CAE 


Stakeholder  Analysis  for  C-IED  T raining  Courses 


3.3.2. 1  High  Level  Requirements 

In  the  context  of  the  IEDD  course,  the  high  level  requirements  for  the  evaluation  module 
are  as  follows: 

•  Consolidate  measurements  of  the  student  activities: 

o  Answers  received; 
o  Locations  visited; 
o  Observations  noted; 
o  Deductions  made; 

o  Eye-tracking  information  (e.g.,  x,  y  position  and  dwell  time);  and, 
o  Hear  Rate  Variability  measurement. 

•  Produce  observations  that  could  have  been  made  (using  eye-tracking  data); 

•  Correlate  the  heart  rate  variability  information  with  the  other  observations  to 
make  deductions  about  the  student  state; 

•  Produce  deductions  that  could  be  made  from  observations,  information  and 
domain  knowledge; 

•  Compare  logical  deductions  to  student  deductions;  and, 

•  Update  the  student  model. 

3. 3.2.2  Relationship  with  Other  Modules 

The  evaluation  module  receives  information  from  the  Training  Delivery  module  and 
produces  information  that  is  used  to  update  the  Student  Model.  Key  inputs  of  the 
evaluation  module  include: 

•  Eye-tracker  data; 

•  Heart  rate  variability  measurement  data; 

•  Student  location  data; 

•  Student  answers  received; 
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•  Student  observations  noted;  and, 

•  Student  deductions. 

Key  outputs  of  the  Evaluation  Module  include: 

•  Student  observation  accuracy; 

•  Student  observation  timeline; 

•  Student  deduction  accuracy;  and, 

•  Student  deduction  timeline. 


3. 3.2. 3  Recommended  Implementation 

The  Evaluation  Module  has  two  key  functions.  The  first  is  the  correlation  of  the  direct 
measures  of  the  student  (such  as  eye-tracking  data)  with  indirect  observations  (such  as 
the  location  of  the  student’s  avatar  in  the  3D  world.  This  task  is  very  specific  to  the  eye¬ 
tracking  system  selected,  and  the  3D  environment  selected.  The  implementation  of  this 
functionality  is  largely  expected  to  be  custom  development. 

The  second  purpose  of  the  Evaluation  Module  is  to  compare  the  deductions  the  student 
has  made  to  the  deductions  the  student  could  have  made.  Description  Logic  (DL)  can 
model  rational  decision  making  and  inference.  Knowledge  (facts  and  rules)  in 
represented  in  ontologies  provide  a  basis  for  making  deductions.  The  evaluation  module 
will  make  use  of  the  ontology  of  IEDD  knowledge,  in  conjunction  with  the  observations 
that  the  student  has  made,  to  produce  inferences  on  the  deductions  the  student  could 
logically  have  produced.  The  comparison  of  the  logical  deductions  with  the  student’s 
deductions  produces  an  evaluation  of  the  student’s  current  state.  The  commercial  DL 
reasoner,  RacerPro4  is  recommended  for  this  component  of  the  evaluation  module. 


3.3.3  Adaptation  Module 

The  purpose  of  the  adaptation  module  is  to  modify  the  delivery  of  training  based  on  the 
evolution  of  the  student  model.  Mechanisms  of  adaptation  should  include: 

•  Adapting  the  pace  of  information  delivery; 

•  Adapting  the  complexity  of  information  delivery; 


4 


The  cost  of  RacerPro  is  between  $2k  and  $10k  depending  on  the  license. 
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•  Adapting  the  learning  style  focus;  and 

•  Altering  the  selection  of  training  content  objects  that  are  presented. 

The  adaptation  module  should  offer  a  range  of  authority  between  adaptive  systems  (i.e., 
system-initiated  adaptation)  and  adaptable  systems  (i.e.,  instructor-initiated  adaptation). 
In  some  cases,  the  adaptation  module  should  also  provide  advice  to  the  instructor.  For 
example,  the  adaptation  module  might  suggest  how  the  learning  experience  might  be 
adapted  to  suit  the  specific  needs  of  a  student. 

The  adaptation  module  should  accommodate  the  adaptation  that  is  possible  with  the 
training  content  available  for  presentation.  For  example,  if  the  training  content  in  use 
does  not  provide  multiple  objects  presenting  similar  information  in  different  learning 
styles,  adaptation  based  on  learning  style  may  not  be  possible. 

The  adaptation  module  is  comprised  of  three  functional  components.  First,  the 
adaptation  assessment  component  is  responsible  for  producing  system-initiated 
adaptation  demands.  Second,  the  instructor  control  component  is  responsible  for 
handling  instructor-initiated  adaptation  demands  from  the  instructor  module.  Finally,  the 
Adaptation  engine  must  consider  the  adaptation  demands,  the  available  training  content 
objects  and  the  possible  training  object  adaptations  described  in  the  metadata.  From 
this,  the  adaptation  engine  controls  the  training  delivery  module  in  its  delivery  of  the 
adapted  content. 

The  adaptation  assessment  component  must  examine  the  student  model  and  the  expert 
model  to  produce  adaptation  demands.  Some  adaptation  aspects  will  depend  only  on 
the  student  model.  For  instance,  workload  and  learning  style  are  two  factors  that 
influence  adaptation,  which  depend  only  on  the  student  model.  Other  adaptation 
demands  will  be  based  on  the  differences  between  the  student  model  and  the  expert 
model.  For  instance,  competency  is  assessed  by  the  comparison  of  the  student 
performance  to  expert  performance  and  to  performance  standards  captured  in  the 
expert  model. 

During  instructor-led  or  instructor  monitored  training,  the  instructor  may  produce 
adaptation  demands.  The  instructor  control  component  coordinates  these  demands 
from  the  instructor  module  to  the  adaptation  engine. 

The  adaptation  engine  is  responsible  for  the  translation  of  adaptation  demands  into 
changes  in  presented  training  content.  First,  the  system-initiated  and  instructor-initiated 
adaptation  demands  must  be  balanced  to  place  the  system  response  in  the  desired 
range  of  the  spectrum  of  adaptation  (see  Figure  12).  Next,  the  consolidated  adaptation 
demands  must  be  compared  to  the  available  training  content  objects.  Based  on  the 
metadata  supplied  with  the  training  content  objects,  the  adaptation  engine  then  controls 
the  delivery  of  training  objects  by  the  training  delivery  module,  causing  the  delivery  of 
training  to  be  adapted  to  the  student.  Based  on  the  capabilities  of  the  training  content 
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objects,  this  adaptation  could  be  continuous  during  the  execution  of  a  training  object,  or 
there  could  be  dynamic  alteration  of  the  sequence  of  training  objects  that  are  presented 
to  the  user. 

The  adaptation  engine  must  be  capable  of  utilizing  the  existing  metadata  that 
accompanies  SCORM  compliant  SCOs.  However,  due  to  the  limitations  of  SCORM, 
SCOs  are  a  subset  of  the  possible  training  content  objects  that  this  system  must 
accommodate.  As  this  stream  of  work  will  be  developing  the  format  of  the  other  training 
content  objects  (such  as  virtual  simulation  scenarios),  the  development  of  the  required 
metadata  for  these  objects  should  be  aligned  with  SCORM  2004  and  the  evolution  of 
SCORM  2.0  where  possible. 


Figure  3-7:  Notional  architecture  of  the  adaptation  module 

The  electronic  IEDD  course  design  process  will  decompose  the  training  material  into  a 
number  of  training  content  objects.  Some  of  these  will  be  SCOs,  while  some  of  these 
will  be  resources,  models  and  scenarios  for  use  in  various  simulation  environments.  A 
baseline  course  structure  will  be  produced  that  sequences  these  training  content 
objects  into  a  teachable  course. 

When  the  training  content  objects  are  hosted  in  the  adaptive  learning  framework,  a 
range  of  adaptation  options  will  then  be  possible  during  training  sessions.  The 
adaptation  module  can  adapt  the  sequence  of  training  content  object  presentation,  the 
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complexity  of  simulation  scenarios  and  the  pace  of  simulation  scenarios  based  on 
student  measures  and  instructor  adaptation  requests. 

Further,  the  training  content  objects  can  be  hosted  in  an  extended  e-learning 
environment,  such  as  the  AFIILE  LMS.  Extensions  to  the  LMS  environment  will  be 
required  to  handle  simulation-based  content,  which  will  not  be  SCORM  compliant.  In 
this  environment,  instructor-base  adaptation  is  still  possible  through  variation  of  training 
content  object  sequencing,  and  manual  selection  of  simulation  scenarios  from  a 
‘scenario  library’. 


3.3.3. 1  High  Level  Requirements 

In  the  context  of  the  IEDD  course,  the  high  level  requirements  for  the  Adaptation 
Module  include: 

•  Comparison  of  the  student  observations  and  timeline  to  an  expert  standard  for 
observation; 

•  Comparison  of  the  student  questioning  to  an  expert  standard; 

•  Comparison  of  the  student  deductions  and  timeline  to  an  expert  standard; 

•  Comparison  of  student  errors  to  error  stereotypes; 

•  Selection  of  coaching  material; 

•  Receiving  coaching  instructions  from  the  instructor;  and, 

•  Command  the  training  delivery  module  to  deliver  required  coaching 


3. 3.3.2  Relationship  with  Other  Modules 

The  Adaption  Module  works  with  the  Student  Model  and  the  Expert  Module  to  produce 
commands  to  the  Training  Module.  The  Adaption  Module  also  takes  commands  from 
the  Instructor  module:  Key  inputs  to  the  Adaption  Module  include: 

•  Student  events  timeline; 

•  Expert  representation;  and, 

•  Error  stereotypes. 
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The  output  from  the  Adaption  Module  is  coaching  commands  for  the  Training  Delivery 
Module. 


3. 3. 3. 3  Recommended  Implementation 

The  Adaption  Module  will  compare  the  Student  Model  (output  from  the  Evaluation 
Module)  to  a  model  of  expert  performance  to  produce  a  performance  measure  for  the 
student.  The  Adaption  Module  will  compare  the  Student  Model  to  common  error 
stereotypes  to  potentially  classify  the  student  actions  as  belonging  to  a  stereotypical 
error  class.  Based  on  these  assessments,  the  Adaption  Module  will  select  appropriate 
coaching  material  for  presentation  to  the  student. 

It  is  recommended  that  the  Adaption  Module  uses  the  same  ontology  based  reasoning 
approach  as  the  Evaluation  Module.  RacerPro  is  the  recommended  reasoner.  RacerPro 
is  used  as  a  system  for  managing  semantic  web  ontologies  based  on  Web  Ontology 
Language  (OWL).  The  design  phase  will  confirm  RacerPro’s  licensing  model,  costs  and 
medium  term  capability  to  meet  this  requirement.  At  the  same  time,  the  technology 
watch  task  may  consider  a  short  evaluation  of  RacerPro’s  capability  compared  to  other 
COTS  reasoning  engines.  The  deductive  process  for  selecting  appropriate  coaching 
material  will  be  represented  in  the  adaption  ontology.  The  reasoner  will  then  act  on  this 
ontology  in  conjunction  with  the  Student  and  Expert  models  to  select  appropriate 
coaching  material.  The  design  phase  will  confirm  RacerPro’s  licensing  model,  costs  and 
medium  term  capability  to  meet  this  requirement.  At  the  same  time,  the  technology 
watch  task  may  consider  a  short  evaluation  of  RacerPro’s  capability  compared  to  other 
COTS  reasoning  engines. 


3.3.4  Expert  Model 

The  purpose  of  the  expert  model  is  to  represent  the  knowledge,  skills  and/or  behaviours 
that  embody  the  desired  end  state  of  the  student.  The  contents  of  the  expert  model  are 
course-specific  and  static  during  the  execution  of  a  training  session  -  the  student  and 
instructor  interactions  with  the  system  do  not  alter  the  representation  of  the  training 
objectives.  The  expert  model  must  encapsulate  at  least  two  aspects  of  representation. 
First,  it  must  encapsulate  the  required  proficiency  of  the  student  as  measured  by  the 
various  training  content  objects  of  the  course  -  the  ‘what’  that  the  student  must 
demonstrate  to  indicate  successful  training.  Second,  it  must  represent  expert 
behaviours  and  skills  -  the  ‘how’  that  is  used  by  the  adaptive  learning  framework  to 
provide  feedback  and  adapt  the  training  content  to  accelerate  learning. 

The  SCORM  standard  specifies  how  SCOs  encapsulate  proficiency  measures.  The 
adaptive  learning  framework  must  make  use  of  training  content  objects  that  are  SCOs. 
For  SCOs,  the  SCORM  methodology  for  proficiency  measures  should  be  used  as  the 
basis  of  the  proficiency  standard  of  the  expert  model.  For  training  content  objects  that 
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are  not  SCORM  compliant,  the  SCORM  methodology  for  proficiency  representation 
should  be  followed,  where  SCORM  technologies  can  represent  the  desired  measures  of 
proficiency.  For  any  proficiency  measures  that  cannot  be  represented  within  the 
SCORM  standard,  the  representation  used  in  the  expert  model  should  take  into 
consideration  the  evolving  SCORM  2.0  standard  if  relevant. 

The  representation  of  expert  skills  and  behaviours  will  be  highly  dependent  on  the 
technology  used  to  implement  the  adaptation  module.  The  expert  model  is  the  profile  of 
desired  behaviour,  used  by  the  adaptation  module  to  assess  the  student’s  competency. 
The  expert  skill  and  behaviour  representation  will  also  be  populated  in  a  fashion 
dependant  on  the  adaptation  module  implementation. 

The  expert  model  is  used  by  the  adaptation  engine  to  assess  the  student’s  state  of 
competency  during  training  delivery.  The  student  model  is  compared  to  the  expert 
model  to  assess  the  nature  of  the  student’s  current  deficiency,  to  drive  the  adaptation 
process. 

In  the  context  of  the  IEDD  Operator  Course,  there  are  three  relevant  aspects  to  the 
expert  model.  The  first  aspect  is  a  model  of  rational  decision  making  with  regard  to  lEDs 
and  their  disposal.  The  second  aspect  is  a  representation  of  the  best  practices  for 
obtaining  the  information  required  for  decision  making,  through  observation  of  the 
scene,  questioning  of  witnesses  and  investigation  of  the  scene.  Finally,  the  expert 
model  should  represent  common  errors  and  alternate  learning  paths. 

The  model  of  rational  decision  making  pertaining  to  IEDD,  based  on  argument  theory, 
can  be  represented  by  an  ontology.  An  ontology  is  a  static  data  structure  that 
represents  the  facts  about  lEDs,  and  the  rules  that  allow  deductions  based  on  those 
facts.  As  previously  discussed,  this  static  data  structure  will  be  used  by  the  Evaluation 
and  Adaption  modules  to  reason  with  the  observations  the  student  has  made  to  logically 
deduce  the  conclusions  the  student  should  and  should  not  be  able  to  make.  The 
ontology  of  IEDD  is  an  expert  model  that  allows  the  system  to  produce  the  conclusions 
an  expert  would  be  able  to  deduce  from  a  given  set  of  observations. 

The  second  aspect  of  the  expert  model  must  provide  guidance  as  to  the  priority  and 
sequence  of  actions  an  expert  would  follow  to  produce  the  required  observations.  If  the 
actions  of  an  expert  at  IEDD  can  be  defined  procedurally,  this  information  can  also  be 
represented  as  part  of  the  IEDD  ontology.  If  the  actions  and  priorities  of  an  expert 
operator  are  controlled  by  judgement,  inference  techniques  or  an  artificial 
intelligence/pattern  matching  technique  might  be  required  to  model  this  aspect  of  expert 
behaviour. 

The  World  Wide  Web  Consortium  (W3C)  standardized  Web  Ontology  Language  (OWL) 
is  recommended  as  the  basis  for  the  expert  model.  The  CommonKADS  templates 
provide  guidance  in  the  development  of  such  an  ontology,  and  the  output  of  the 
recommended  processes  can  be  mapped  to  OWL  ontologies. 
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3. 3.4.1  High  Level  Requirements 

In  the  context  of  the  IEDD  Operator  Course,  the  high  level  requirements  for  the  Expert 
Model  are  as  follows: 

•  Represent  decision  making  in  IEDD; 

•  Represent  priorities,  sequences  and  procedures  for  IEDD; 

•  Represent  common  errors  in  IEDD  reasoning;  and, 

•  Represent  alternate  decision  paths  in  IEDD  procedures. 

3. 3.4.2  Relationship  with  Other  Modules 

The  Expert  Model  is  a  static  representation  of  expert  behaviour.  The  Expert  Model  is 
referenced  by  the  Evaluation  Module  and  by  the  Adaption  Module. 


3. 3.4.3  Recommended  Implementation 

For  the  prototype  activity,  it  is  recommended  that  the  Expert  Model  be  restricted  to  the 
aspects  of  proper  IEDD  that  can  be  represented  as  a  structured  ontology.  OWL  is 
recommended  as  the  standard  for  this  ontology,  due  to  the  maturity  of  the  standard,  and 
the  availability  of  tools  to  construct  and  reason  with  OWL  ontologies. 


3.3.5  Student  Model 

The  purpose  of  the  Student  Model  is  to  represent  the  current  knowledge,  skills  and/or 
behaviours  that  embody  the  student.  The  contents  of  the  Student  Model  are  course  and 
student  specific,  and  evolve  dynamically  during  the  execution  of  a  training  session  -  the 
student’s  interactions  with  the  system  updates  the  modelled  state  of  the  student’s 
competency. 

Where  possible,  the  Student  Model  should  be  partitioned  into  course  independent 
information  (such  as  learning  style)  and  course  specific  information  (such  as 
competency).  In  an  enterprise  system,  the  course  independent  information  should  be 
stored  in  such  a  fashion  that  it  can  be  reused  in  other  interactions  with  the  same 
student. 

The  course  dependant  component  of  the  student  model  has  the  same  composition  as 
the  Expert  Model,  other  than  the  fact  that  it  contains  the  current  measure  of  the 
student’s  state  of  competency,  rather  than  the  desired  state. 
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The  course  independent  component  of  the  Student  Model  contains  information  such  as 
learning  style,  user  preferences  and  student  background.  This  aspect  of  the  student 
model  should  be  populated,  where  possible,  with  historic  information  of  the  student, 
such  as  from  an  LMS  or  from  previous  student  interactions.  In  certain  cases,  additional 
assessments  may  be  required  at  the  time  of  training  delivery  to  initialize  this  component 
of  the  student  model.  For  instance,  if  student  learning  style  is  a  dimension  of  adaptation, 
and  no  information  currently  exists  on  the  current  student’s  learning  style,  a  learning 
style  quiz  may  need  to  be  delivered  as  training  preliminaries.  Developing  or  refining  this 
information  using  the  measures  of  the  student’s  interaction  with  the  course  specific 
learning  content  should  be  considered  where  possible. 

The  Student  Model  is  updated  during  training  execution  by  the  Evaluation  Module.  The 
student’s  interactions  with  the  system  and  performance  in  training  content  driven 
evaluations  are  used  to  update  the  training  course  specific  aspects  of  the  Student 
Model  dynamically. 

The  Student  Model  is  used  by  the  Adaptation  Module.  The  adaptation  engine  compares 
the  current  Student  Model  with  the  Expert  Model  to  assess  the  student’s  deficiencies. 
This  information  is  used  to  drive  the  adaptation  process. 


3.3.5. 1  High  Level  Requirements 

In  the  context  of  the  IEDD  Operator  Course,  the  high  level  requirements  for  the  Student 
Model  are  as  follows: 

•  Represent  the  observations  the  student  has  made; 

•  Represent  the  actions  the  student  has  taken; 

•  Represent  the  decisions  the  student  has  made; 

•  Represent  the  learning  style  of  the  student; 

•  Represent  the  physiological  state  of  the  student  (e.g.,  HRV  data);  and, 

•  Represent  the  attentional  focus  of  the  student  (e.g.,  eye-tracking  and  interaction 
device  data;  such  as  keystrokes  and  mouse  clicks). 
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3. 3. 5. 2  Relationship  with  Other  Modules 

The  Student  Model  is  a  dynamic  store  of  information  about  the  student.  This  data  store 
is  updated  by  the  Evaluation  Module,  and  is  accessed  by  both  the  Evaluation  and 
Adaption  Modules. 


3. 3. 5. 3  Recommended  Implementation 

Due  to  the  dynamics  updates  of  the  Student  Model,  and  the  potentially  asynchronous 
access  of  the  Student  Model  between  both  the  Expert  and  Adaption  Modules,  a 
relational  database  implementation  is  suggested  for  the  Student  Model.  Based  on  the 
information  systems  emphasis  of  the  Adaption  Module,  Evaluation  Module  and  any 
potential  LMS,  it  is  suggested  that  the  implementation  tool  selection  for  the  Student 
Model  should  be  deferred  until  the  detailed  architecture  is  defined  and  the 
compatibilities  and  supported  standards  of  the  applications  that  must  integrate  with  the 
student  model  are  known. 


3.3.6  Experiment  Control  Module 

The  purpose  of  the  Experiment  Control  Module  is  to  control  the  execution  of  the 
adaptive  learning  framework  in  a  controlled  fashion  that  collects  the  required  data.  The 
Experiment  Control  Module  will,  at  minimum,  be  required  to  provide  initialization  data 
(such  as  student  history)  as  required  by  the  various  modules,  and  to  collect 
experimental  data  as  required  from  the  various  modules. 

The  Experiment  Control  Module  performs  a  role  similar  to  the  LMS  in  an  enterprise 
training  implementation.  A  modern  LMS,  such  as  the  AFIILE  LMS  (e.g.,  Saba  Learn), 
should  be  considered  for  this  role.  However,  the  capabilities  of  the  LMS  need  to  be 
considered  from  the  perspectives  of  both  supporting  the  experimental  goals,  and  the 
advanced  features  of  the  training  content  and  the  adaptive  learning  framework. 

It  is  expected  that  the  experimenter  will  be  able  to  make  use  of  Blackboard  functionality, 
as  outlined  in  Section  3.3.8.  The  Experiment  Control  Module  should  include  a 
Experimenter’s  Blackboard  to  present  the  experimenter  with  relevant  information. 


3.3.6. 1  High  Level  Requirements 

In  the  context  of  the  IEDD  Operator  Course,  the  high  level  requirements  for  the 
Experiment  Control  Module  are  as  follows: 

•  Control  the  execution  of  the  training  system; 
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•  Monitor  the  execution  of  the  training  system; 

•  Log  relevant  data  for  later  analysis;  and, 

•  Enable  injection  of  experimenter  comments  and  notes  into  the  data  log. 


3. 3.6.2  Relationship  with  Other  Modules 

The  Experiment  Control  Module  controls  and  monitors  the  execution  of  all  of  the 
modules  of  the  system.  It  is  expected  that  data  for  the  experiment  log  will  be  drawn  from 
all  of  the  other  modules  in  the  system. 


3. 3.6. 3  Recommended  Implementation 

Although  it  is  anticipated  that  some  kind  of  LMS  will  be  used  to  implement  the 
Experiment  Control  Module,  it  is  suggested  that  the  LMS  tool  selection  for  this  module 
should  be  deferred  until  the  detailed  architecture  is  defined  and  the  compatibilities  and 
supported  standards  of  the  applications  that  must  integrate  with  the  Experiment  Control 
Module  are  known. 


3.3.7  Instructor  Module 

The  purpose  of  the  Instructor  Module  is  to  allow  the  instructor  to  interact  with  the 
student(s)  when  requested  by  the  student,  desired  by  the  instructor  or  required  by  the 
training  content.  In  this  regard,  the  hardware  and  software  configuration  of  the  instructor 
module  will  be  similar  to  the  training  delivery  module.  The  instructor  must  be  able  to 
monitor  or  participate  in  the  simulation-based  training  content  to  perform  instructor-led 
or  monitored  training  in  these  environments.  As  well,  the  instructor  must  be  capable  of 
collaborating  and  interacting  with  the  student  in  an  unstructured  fashion  to  accomplish 
instructor-led  training  delivery. 

The  Instructor  Module  extends  the  training  delivery  module  in  that  the  instructor  may  be 
required  to  assess  the  student’s  performance  with  certain  types  of  training  content,  and 
the  instructor  must  be  capable  of  executing  instructor-initiated  adaptation  requests.  The 
recommended  implementation  of  the  Instructor  Module  is  a  training  delivery  module  with 
access  to  an  instructor’s  interface  of  the  Experiment  Control  Module.  Finally,  an 
Instructor  Module  will  be  required  for  each  instructor  involved  in  the  training  session. 

It  is  expected  that  the  instructor  will  be  able  to  make  use  of  Blackboard  functionality,  as 
outlined  in  Section  3.3.8.  The  Instructor  Module  should  include  an  Instructor’s 
Blackboard  to  present  the  instructor  with  relevant  information. 
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3.3.7. 1  High  Level  Requirements 

In  the  context  of  the  IEDD  Operator  Course,  the  high  level  requirements  for  the 
Instructor  Module  are  as  follows: 

•  Monitor  the  performance  of  the  student; 

•  Inject  coaching  stimulus  to  the  student; 

•  Inject  scenario  events  into  the  training  system;  and, 

•  Control  the  progression  of  the  scenario  (such  as  pause  functionality). 


3. 3.7.2  Relationship  with  Other  Modules 

The  Instructor  Module  will  monitor  information  from  the  Training  Delivery,  Evaluation 
and  Adaption  Modules  in  a  fashion  similar  to  the  Experiment  Control  Module.  The 
Instructor  Module  will  also  interact  with  the  Adaption  Module  to  initiate  instructor-led 
adaptations  or  instructor  initiated  coaching  sessions. 


3. 3.7. 3  Recommended  Implementation 

Although  it  is  anticipated  that  some  kind  of  LMS  will  be  used  to  implement  the  Instructor 
Module,  it  is  suggested  that  the  LMS  tool  selection  for  this  module  should  be  deferred 
until  the  detailed  architecture  is  defined  and  the  compatibilities  and  supported  standards 
of  the  applications  that  must  integrate  with  the  Instructor  Module  are  known. 


3.3.8  Blackboard 

A  blackboard  system  is  an  artificial  intelligence  application  based  on  the  blackboard 
architectural  model,  where  a  common  knowledge  base,  the  "blackboard",  is  iteratively 
updated  by  a  diverse  group  of  specialist  knowledge  sources,  starting  with  a  problem 
specification  and  ending  with  a  solution.  Each  knowledge  source  updates  the 
blackboard  with  a  partial  solution  when  its  internal  constraints  match  the  blackboard 
state.  In  this  way,  the  specialists  work  together  to  solve  the  problem.  The  blackboard 
model  was  originally  designed  as  a  way  to  handle  complex,  ill-defined  problems,  where 
the  solution  is  the  sum  of  its  parts. 

A  blackboard-system  application  consists  of  three  major  components: 

1 .  The  software  specialist  modules,  which  are  called  knowledge  sources  (KSs).  Like 
the  human  experts  at  a  blackboard,  each  knowledge  source  provides  specific 
expertise  needed  by  the  application.  The  ability  to  support  interaction  and 
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cooperation  among  diverse  KSs  creates  enormous  flexibility  in  designing  and 
maintaining  applications.  As  the  pace  of  technology  has  intensified,  it  becomes 
ever  more  important  to  be  able  to  replace  software  modules  as  they  become 
outmoded  or  obsolete. 

2.  The  blackboard,  a  shared  repository  of  problems,  partial  solutions,  suggestions, 
and  contributed  information.  The  blackboard  can  be  thought  of  as  a  dynamic 
"library"  of  contributions  to  the  current  problem  that  have  been  recently 
"published"  by  other  knowledge  sources. 

3.  The  control  shell,  which  controls  the  flow  of  problem-solving  activity  in  the 
system.  Within  a  blackboard  system,  KSs  need  a  mechanism  to  organize  their 
use  in  the  most  effective  and  coherent  fashion.  In  a  blackboard  system,  this  is 
provided  by  the  control  shell. 


3.3.8. 1  High  Level  Requirements 

In  the  context  of  the  IEDD  Operator  Course,  the  high  level  requirements  for  the 
Blackboard  are  as  follows: 

•  A  library  of  the  available  knowledge  in  the  environment  (i.e.,  SA  elements) 
accessed  by: 

o  Direct  questioning  of  NPCs  within  the  simulated  environment;  and, 

o  Direct  observation  of  cues  (e.g.  disturbance  of  road  surface)  within  the 
simulated  environment; 

•  The  knowledge  required  to  make  each  decision  with  the  threat  assessment 
process; 

•  The  decisions  required  to  complete  the  threat  assessment  process; 

•  Track  the  knowledge  acquisition  and  decision  making  progress  of  the  student  in 
relation  to  a  benchmark; 

•  Track  the  affective  state  (e.g.,  HRV  data)  of  the  student  in  relation  to  relevant  SA 
elements;  and, 

•  Track  the  eye  fixation  point  and  dwell  time  in  relation  to  relevant  SA  elements. 
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Figure  3-8  provides  an  example  of  a  blackboard  which  tracks  (in  real  time)  the  following 
information  pertaining  to  the  student  using  the  notional  IEDD  ITS  user  experience  as 
described  in  Section  3. 2. 2.4: 

•  What  knowledge  in  the  environment  (i.e.,  SA  elements)  has  been  acquired  by  the 
student  (or  not)  from  direct  questioning  of  NPCs  within  the  simulated 
environment  (performance  tracking); 

•  What  knowledge  in  the  environment  (i.e.,  SA  elements)  has  been  acquired  by  the 
student  (or  not)  from  direct  observation  of  cues  (e.g.  disturbance  of  road  surface) 
within  the  simulated  environment  (performance  tracking); 

•  Current  threat  assessment  of  the  student  in  terms  of  which  type  of  IED  the 
student  believes  it  to  be  (performance  tracking); 

•  Current  affective  state  of  the  student; 

•  Past  affective  state  of  the  student  in  relation  to  relevant  SA  elements; 

•  Current  eye  fixation  point  and  dwell  time;  and, 

•  Past  eye  fixation  point  and  dwell  time  in  relation  to  relevant  SA  elements. 


3. 3. 8. 2  Relationship  with  Other  Modules 

As  discussed  in  the  previous  two  sections,  it  is  anticipated  that  the  Instructor  and 
Experiment  Control  Modules  will  both  benefit  from  Blackboard-type  functionality. 


3.3.8. 1  Recommended  Implementation 

Previous  research  conducted  under  the  auspices  of  the  United  Kingdom’s  “Cognitive 
Cockpit”  technology  demonstration  program  (for  a  review  see  Banbury  et  al. ,  2007) 
developed  a  blackboard  architecture  to  satisfy  similar  adaptation-based  requirements  to 
those  of  the  IEDD  ITS.  The  blackboard  was  implemented  in  an  html  format  which  all 
other  modules  were  able  to  access.  We  suggest  that  the  project  team  consult  with 
experts  from  QinetiQ  (Dr  Blair  Dickson)  for  advice  on  how  the  blackboard  might  be 
implemented  for  the  IEDD  ITS. 
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ITS  Blackboard 


STUDENT  STATUS 

Student  ID 

12345678 

Learning  Style 

Visual  dominant 

Content-Vis#  1 

Affective  State  (/50) 

45 

High 

Total  %  of  SA  acquired 

General  -  70% 

Critical  -  60% 

Threat  Assessment 

Time-operated  IED 

Incorrect 

SCENARIO  STATUS 

Scenario  type 

Victim-operated  IED 

Time  elapsed 

HH:MM:SS 

Current  NPC  Questioned 

OCS 

NPC  Questioned  (Total) 

2  of  3 

Data  File 

12345678 data.xls 

NPC  #1  QUESTIONING  STATUS 

NPC  #1 

ANA  Officer 

Questioned  (Freq) 

Yes  (1) 

SA  Element 

Description 

Acq 

Acquired  SA  Element  #la 

Wedding  guests  arrived 

Yes 

Acquired  SA  Element  #lb 

No  cell  phone  activity 

No 

Acquired  SA  Element  #lc 

No  recent  attacks 

Yes 

Acquired  SA  Element  #ld 

Decrease  in  attacks 

Yes 

Acquired  SA  Element  #le 

Weddings  attacked 

Yes 

Acquired  SA  Element  #lf 

Civilians  not  attacked 

No 

Acquired  SA  Element  #lq 

High  freq.  of  ANA  partrols 

Yes 

Critical  S A  Element 

Description 

Acq 

Acquired  SA  Element  #lh 

Roadside  parking  habits 

Yes 

Acquired  SA  Element  #li 

Wedding  guests  arrived 

Yes 

Acquired  SA  Element  #lj 

ANA  from  rival  tribe 

No 

NPC  #1  INSTRUCTIONAL  STATUS 

Main  Teaching  Point 

Description 

Freq 

Intervention 

MTP  #la 

Closed  question 

0 

MTP  #lb 

Cultural  sensitivity 

2 

CoachScript  MTP  #lb 

MTP  #lc 

Insufficient  critique  of  answer 

1 

CoachScript  MTP  #lc 

MTP  #ld 

Aware  of  importance 

0 

MTP  #le 

Leading  question 

4 

CoachScript  MTP  #ld 

MTP  #lf 

Not  listening 

0 

MTP  #lg 

Jargon 

2 

CoachScript  MTP  #lf 

MTP  #lh 

Negative  question 

3 

CoachScript  MTP  #lg 

MTP  #li 

Not  relevant 

1 

HintScript  MTP  #lb 

MTP  #lj 

All  SA  elements  acquired 

No 

HintScript  MTP  #lc 

NPC  #1  STUDENT  QT  PERFORMANCE 

%  of  SA  Elements  acquired 

70% 

%  of  Critical  SA  Elements  acquired 

66% 

#  of  Questioning  Errors 

13 

NPC  #2  INSTRUCTIONAL  STATUS 

Main  Teaching  Point 

Description 

Freq 

Intervention 

MTP  #2a 

Closed  question 

0 

MTP  #2b 

Cultural  sensitivity 

0 

MTP  #2c 

Insufficient  critique  of  answer 

1 

CoachScript  MTP  #2c 

MTP  #2d 

Aware  of  importance 

0 

MTP  #2e 

Leading  question 

0 

MTP  #2f 

Not  listening 

0 

MTP  #2g 

Jargon 

1 

CoachScript  MTP  #2f 

MTP  #2h 

Negative  question 

1 

CoachScript  MTP  #2g 

MTP  #2i 

Not  relevant 

1 

HintScript  MTP  #2b 

MTP  #2j 

All  SA  elements  acquired 

No 

HintScript  MTP  #2c 

NPC  #2  QUESTIONING  STATUS 

NPC  #2 

OSC 

Questioned  (Freq) 

In  progress  (0) 

SA  Element 

Description 

Acq 

Acquired  SA  Element  #2a 

Recent  CF  activity  displaced  ins 

Yes 

Acquired  SA  Element  #2b 

No  cell  phone  activity 

No 

Acquired  SA  Element  #2c 

No  recent  attacks 

Yes 

Acquired  SA  Element  #2d 

Decrease  in  attacks 

Yes 

Acquired  SA  Element  #2e 

No  EW  activity 

Yes 

Acquired  SA  Element  #2f 

Roadside  culvert  present 

Yes 

Acquired  SA  Element  #2g 

High  freq.  of  ANA  partrols 

Yes 

Critical  SA  Element 

Description 

Acq 

Acquired  SA  Element  #2h 

IEDD  operators  targeted 

Yes 

Acquired  SA  Element  #2i 

Insurgency  effectiveness  down 

Yes 

Acquired  SA  Element  #2j 

ANA  from  rival  tribe 

No 

Acquired  SA  Element  #2k 

ANA  corruption 

Yes 

NPC  #2  STUDENT  QT  PERFORMANCE 

%  of  SA  Elements  acquired 

90% 

%  of  Critical  SA  Elements  acquired 

50% 

#  of  Questioning  Errors 

4 

NPC  #3  QUESTIONING  STATUS 

NPC  #3 

Trooper  Smith  (Witness  #1) 

Questioned  (Freq) 

No 

SA  Element 

Description 

>40? 

Acquired  SA  Element  #3a 

No  wire  seen 

Acquired  SA  Element  #3b 

No  road  surface  disturbance 

Acquired  SA  Element  #3c 

No  reciever 

Acquired  SA  Element  #3d 

Roadside  culvert  present 

Critical  SA  Element 

Description 

Acq 

Acquired  SA  Element  #3e 

Vantage  point  unused 

NPC  #3  INSTRUCTIONAL  STATUS 

Main  Teaching  Point 

Description 

Freq 

Intervention 

MTP  #la 

Closed  question 

MTP  #lb 

Cultural  sensitivity 

MTP  #lc 

Insufficient  critique  of  answer 

MTP  #ld 

Aware  of  importance 

MTP  #le 

Leading  question 

MTP  #lf 

Not  listening 

MTP  #lg 

Jargon 

MTP  #lh 

Negative  question 

MTP  #li 

Not  relevant 

MTP  #lj 

All  SA  elements  acquired 

No 

NPC  #3  STUDENT  QT  PERFORMANCE 

%  of  SA  Elements  acquired 

%  of  Critical  SA  Elements  acquired 

#  of  Questioning  Errors 

VISUAL  SA  ELEMENTS  OBSERVED 

5-4  Element 

Description 

Eye  Tracker 

Affective  Response 

QT  Confirmation 

Acquired  SA  Element  VI 

No  disturbance  of  road  surface 

Yes 

No 

Yes 

Acquired  SA  Element  V2 

No  disturbance  of  surface  by  side  of  road 

No 

No 

No 

Acquired  SA  Element  V3 

No  evidence  of  wire  in  vicinity 

Yes 

No 

No 

Acquired  SA  Element  V4 

Rock  pile  by  side  of  road 

Yes 

No 

No 

Acquired  SA  Element  V5 

Roadside  culvert  present 

Yes 

Yes 

Yes 

Acquired  SA  Element  V6 

Observation  /  vantage  point  present 

No 

No 

No 

Acquired  SA  Element  V7 

No  evidence  of  timing  device  in  vicinity 

Yes 

No 

Yes 

Figure  3-8:  Notional  contents  of  the  ITS  Blackboard 
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3.3.9  Summary  of  Technological  Requirements  for  the  IEDD  ITS 

Table  3-1  summarises  the  technologies  required  to  meet  the  notional  design  of  the 
IEDD  ITS  described  in  this  report.  In  addition,  any  associated  actions  on  the  part  of  the 
CAE  PS  team  to  implement  those  technologies  are  described. 

Table  3-1:  Summary  of  Technological  Requirements  for  the  IEDD  ITS 


IEDD  ITS 
Component 

Technology  Requirement 

Action  Plan 

Training  Delivery 

XHTML,  Flash,  JavaScript,  VBS2, 

Ensure  licenses  are  in 

Module 

nGrain,  Alelo 

place  /  evaluate  Aleo 
integration 

Evaluation  Module 

Eye-tracking 

Purchase  EasyGaze 

HRV  measurement  device 

Purchase  HRVlive! 

Adaptation  Module 

RacerPro 

Evaluate  alternative  tools  / 
acquire  license 

Expert  Model 

OWL  compliant  ontology 

Use  CommonKADS 
templates  for  guidance 

Student  Model 

LMS 

Evaluate  LMS  options 

Experiment  Control 
Module 

LMS 

Evaluate  LMS  options 

Instructor  Module 

LMS 

Evaluate  LMS  options 

Blackboard 

HTML 

Seek  implementation 
advice  from  QinetiQ 

3.4  Recommended  Implementation  Plan  for  the  IEDD  ITS 

The  following  section  describes  the  tasks  and  schedule  for  the  implementation  and 
evaluation  of  the  IEDD  ITS. 


3.4.1  Identification  of  IEDD  ITS  Development  Route-Map 

Banbury  et  al. ,  (2007)  developed  guidance  for  the  design  and  development  of  an 
Intelligent  Adaptive  System  (IAS)  based  on  information  gathered  during  literature  review 
activities  about  theoretical  frameworks,  analytical  approaches,  multi-agent  systems,  and 
the  use  of  psychophysiological-  and  behaviour-based  feedback.  Their  report  highlighted 
the  strengths  and  weaknesses  of  several  design  and  analysis  approaches,  and  created 
a  development  route-map  for  the  successful  development  of  an  IAS  (see  Figure  3-9). 
The  first  step  is  to  conduct  a  taxonomic  analysis  of  the  proposed  system,  followed  by 
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the  selection  of  the  appropriate  framework,  analysis  methodology,  and  design 
methodology.  The  final  step  is  the  selection  of  the  appropriate  design  guidelines  (in 
terms  of  principles  of  adaptation,  interaction,  etc.). 

The  following  section  describes  each  step  in  more  detail,  with  particular  emphasis  on 
the  selection  of  the  most  appropriate  tools  and  methodologies  for  the  IEDD  ITS  using 
the  decision  trees  developed  by  Banbury  et  al.,  2007. 


Figure  3-9:  Development  route-map  for  Intelligent  Adaptive  Systems  (from  Banbury, 

Gauthier  and  Scipione,  2007) 


3.4.1 .1  Conduct  Taxonomic  Analysis 

Banbury  et  al.,  (2007)  suggest  that  the  development  and  implementation  of  intelligent 
adaptive  systems  can  be  guided  by  a  taxonomic  approach  that  scopes  the  options 
available  for  the  capability  and  functionality  of  the  system.  In  addition,  a  taxonomic 
approach  can  assist  the  creation  of  an  audit  trail  for  the  design  of  the  system.  Finally,  it 
can  provide  a  road-map  for  development  in  that  it  allows  the  development  team  to  focus 
on  specific  implementations  after  scoping  all  of  the  possibilities.  Figure  3-10  describes  a 
taxonomic  analysis  approach  tailored  to  the  requirements  of  the  IEDD  ITS. 
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I 

LEARNING  POINT  DESCRIPTION 

I 


SCENARIO 

TIMELINE 


Figure  3-10:  Notional  taxonomic  analysis  framework  for  IEDD  Operator  ITS 

In  order  to  create  the  taxonomy  for  the  IEDD  ITS,  the  following  factors  need  to  be 
defined: 


■  The  learning  objectives  (i.e. ,  Main  Teaching  Points;  MTPs)  for  each  learning 
point  in  the  scenario.  The  learning  objectives  can  be  broken  down  into  the 
knowledge,  skills  and  attitudes  required  to  be  judged  as  competent  in  that 
learning  point; 

■  The  instructional  intervention  required  to  teach  or  evaluate  student  competence 
for  learning  point.  The  instructional  interventions  can  be  broken  down  into 
adaptive  learning  (i.e.,  hinting  or  influencing)  and  intelligent  tutoring  (i.e.,  explicit 
coaching).  The  choice  of  instructional  intervention  should  be  determined  by 
theories  of  learning  and  pedagogy.  To  date,  traditional  learning  theories  and 
educational  paradigms  have  continued  to  be  used  to  support  distance  learning. 
As  learning  technologies  mature  beyond  page-turning  Computer  Based  Training 
(CBT)  and  video  demonstrations,  a  variety  of  learning  approaches  will  have  to  be 
understood  and  integrated  to  achieve  learning  objectives  and  outcomes.  This 
part  of  the  analysis  will  briefly  review  learning  theories  that  have  been  applied  to 
novel  learning  technologies. 

■  The  technological  requirements  needed  to  implement  the  instructional 
interventions  identified  for  each  scenario  learning  points.  These  interventions 
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should  include  the  mechanisms  for  adaptation  (e.g.,  eye  tracking, 
psychophysiological  response  etc).  Once  again,  the  technological  requirements 
can  be  broken  down  into  adaptive  learning  and  intelligent-tutoring  technologies. 

■  The  requirements  for  evaluating  the  utility  of  the  adaptive  learning  and  intelligent 
tutoring  technologies  for  each  learning  point.  These  requirements  should 
describe  the  measures  of  performance  and  the  tools  used.  Once  again,  the 
evaluation  requirements  can  be  broken  down  into  adaptive  learning  and 
intelligent-tutoring  technologies. 

In  defining  these  factors,  the  taxonomic  approach  allows  the  instructional  design  to  be 
explicitly  identified  for  each  learning  point  within  the  scenario,  and  through  a  specific 
adaptive  learning  or  intelligent  tutoring  technology.  The  approach  also  allows  the 
development  team  to  construct  an  appropriate  mission  scenario  which  encompasses 
the  range  of  system  functionality  and  capability  identified  by  the  taxonomy.  The  mission 
scenario  is  used  in  both  the  subsequent  analysis  (i.e. ,  as  a  precursor  to  the  functional 
decomposition  of  tasks,  goals  and/or  functions)  and  verification  (i.e.,  determination  of 
measures  of  effectiveness  and  performance)  activities.  Finally,  the  taxonomic  approach 
allows  the  development  team  to  quickly  scope  the  functionality  and  capability  of  the 
IEDD  ITS  in  terms  priority  and  feasibility.  This  allows  the  development  team  to  maximise 
the  impact  of  the  IEDD  ITS  on  instructional  operational  performance  whilst  reducing 
development  risk  (e.g.,  due  to  dependence  on  immature  technology)  within  the  time  and 
budgetary  constraints  of  the  project. 

We  recommend  that  the  taxonomic  analysis  be  conducted  early  during  the  next  phase 
of  the  project  in  order  to  support  the  implementation  of  ITS  technologies  within  the  IEDD 
Operator  Course. 


3.4.1. 2  Select  Framework 

The  selection  of  an  appropriate  development  framework  affords  the  development  team 
a  number  of  advantages:  reduction  in  development  time  and  costs  from  leveraging 
previous  research;  benefit  from  the  lessons  learnt  from  past  projects;  and  providing  an 
insight  into  the  potential  operational  impact  of  the  developed  system.  One  of  the  most 
recent  and  comprehensive  attempts  to  generate  a  design  and  development  framework 
for  IASs  was  by  Edwards  (2004,  and  see  Section  9.3.3  of  Banbury  et  al.’s  report). 
Edwards  examined  a  variety  of  theoretical  approaches  to  generate  a  generic,  integrated 
and  comprehensive  framework  for  the  development  of  an  intelligent,  adaptive,  agent- 
based  system  for  Uninhabited  Air  Vehicle  (UAV)  control. 

We  recommend  the  use  of  Edward’s  framework  to  support  the  development  of  the  IEDD 
ITS.  For  example,  we  have  already  recommended  the  use  of  the  CommonKADS 
methodology,  as  recommended  by  Edwards,  for  guiding  the  development  of  the  Expert 
Model. 
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In  addition,  Banbury  et  al.,  (2007)  provide  advice  for  the  selection  of  specific 
frameworks  that  may  be  used  to  guide  the  development  process  for  a  particular  target 
domain  (e.g.,  lessons  learnt,  appraisal  of  technological  maturity).  Their  decision  tree  for 
the  selection  of  design  frameworks  was  completed  for  the  proposed  IEDD  ITS  (see 
Figure  3-1 1 ) 


Figure  3-1 1 :  Decision  tree  for  selection  of  design  frameworks  applicable  to  the  IEDD  ITS 

(from  Banbury,  Gauthier  and  Scipione,  2007) 

We  recommend  that  we  take  a  closer  look  at  the  intelligent  adaptive  framework 
developed  under  the  auspices  of  the  United  Kingdom’s  “Cognitive  Cockpit”  technology 
demonstration  program  (for  a  review  see  Banbury  et  al.,  2007).  For  example,  this 
project  developed  a  blackboard  architecture  to  satisfy  similar  adaptation-based 
requirements  to  those  of  the  IEDD  ITS. 
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3.4.1. 3  Select  Analysis  Methodology 

Analysis  methodologies  provide  the  GUI  display  and  control  requirements  needed  for 
the  design  of  the  IAS,  as  well  as  a  functional  decomposition  of  the  tasks  within  the 
domain  envisaged  for  it.  In  addition,  analysis  methodologies  are  also  used  as  part  of  the 
training  needs  analysis  to  develop  the  instructional  content  for  the  IEDD  ITS  and 
populate  the  Expert  and  Student  Models  described  earlier. 

Banbury  et  al,  (2007)  provide  advice  for  the  selection  of  analysis  methodologies  based 
upon  project  constraints  that  may  be  used  to  guide  the  development  process  (e.g., 
lessons  learnt,  validation  studies).  Their  decision  tree  for  the  selection  of  analysis 
methodology  was  completed  for  the  proposed  IEDD  ITS  (see  Figure  3-12). 


Figure  3-12:  Decision  tree  for  selection  of  analysis  methodologies  applicable  to  the  IEDD 
ITS  (from  Banbury,  Gauthier  and  Scipione,  2007) 

Given  the  complexity  and  cognitive  nature  of  the  instructional  material  (i.e.,  threat 
assessment  and  decision  making),  the  detailed  knowledge  capture  requirements  to 
implement  an  ITS,  and  the  availability  of  expert  analysts  within  the  CAE  PS  project 
team,  we  recommend  the  utilisation  of  cognitive  task  analysis-based  techniques  to 
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develop  the  instructional  content  for  the  IEDD  ITS  and  populate  the  Expert  and  Student 
Models. 


3.4.1. 4  Select  Design  Methodology 

Design  methodologies  assist  in  the  identification  and  definition  of  the  knowledge 
required  by  intelligent  systems.  Banbury  et  al. ,  (2007)  provide  advice  for  the  selection  of 
design  methodologies  that  may  be  used  to  guide  the  development  process  of  a 
particular  target  system  (e.g.,  lessons  learnt,  validation  studies).  Their  decision  tree  for 
the  selection  of  analysis  methodology  was  completed  for  the  proposed  IEDD  ITS  (see 
Figure  3-13). 


Figure  3-13:  Decision  tree  for  selection  of  design  methodologies  applicable  to  the  IEDD 
ITS  (from  Banbury,  Gauthier  and  Scipione,  2007) 

Given  the  requirement  for  multiple  agents,  feedback,  and  cognitive-based  instructional 
material,  together  with  the  complexity  of  IEDD  scenarios,  we  recommend  the  use  of 
Edward’s  (2004)  Explicit  Models  Design  to  support  the  implementation  of  ITS 
technologies  within  the  IEDD  Operator  Course. 
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3.4.1. 5  Select  Operator  State  Monitoring  Approach 

Operator  state  monitoring  is,  part,  the  mechanism  by  which  the  IEDD  ITS  will  adapt  the 
content  and  presentation  of  the  learning  material  in  order  to  optimise  the  student’s 
learning  experience.  Banbury  et  al.,  (2007)  provide  advice  for  the  selection  of  an 
operator-state  monitoring  approach  that  may  be  used  to  guide  the  development  process 
of  the  target  system  (e.g.,  lessons  learnt,  appraisal  of  technological  maturity).  Their 
decision  tree  for  the  selection  of  analysis  methodology  was  completed  for  the  proposed 
IEDD  ITS  (see  Figure  3-14). 


Figure  3-14:  Decision  tree  for  selection  of  operator  state  monitoring  approaches 
applicable  to  the  IEDD  ITS  (from  Banbury,  Gauthier  and  Scipione,  2007) 
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Kramer  et  al.,  (2010)  provided  an  in-depth  review  of  suitable  psychophysiological-based 
and  behavioural-based  approaches  to  the  monitoring  of  operator  state,  and 
recommended  a  combination  approach  utilising  eye-tracking  and  HRV  measurement  to 
monitor  operator  state  for  the  IEDD  ITS.  The  results  of  the  decision  tree  presented  in 
Figure  3-14  are  consistent  with  their  recommendations. 

We  recommend  that  a  combination-based  approach  to  the  monitoring  of  operator  state; 
given  that  one  of  the  objectives  of  the  ITS  project  is  to  examine  emerging  operator  state 
monitoring  technologies  in  relation  to  their  suitability  to  adaptive  learning  and  intelligent 
tutoring  approaches. 


3.4.2  IEDD  ITS  Implementation  Plan 

This  section  lays  out  a  work  plan  for  implementing  a  working  prototype  of  the  IEDD  ITS 
to  enable  its  evaluation  by  students  of  the  IEDD  Operator  Course.  An  iterative  approach 
is  outlined  to  push  early  integration  of  the  proposed  technologies.  This  enables  early 
evaluation  of  a  minimally  functional  system  by  the  technical  authority.  This  internal 
evaluation  supports  a  more  detailed  prioritization  of  the  functionality  for  the  prototype. 
Due  to  resource  and  schedule  constraints,  this  focus  will  be  necessary  to  direct  the 
prototype  development  effort. 

The  work  plan  defined  in  the  following  sections  is  divided  into  two  iterations.  The  end 
state  of  the  first  iteration  is  a  minimally-functional  prototype  for  evaluation  by  the 
scientific  authority.  The  purpose  of  this  evaluation  is  to  enable  effective  mid-point 
correction  to  the  priorities  of  the  technical  development.  The  end  state  of  the  second 
iteration  is  the  ITS  prototype  that  will  support  the  evaluation  phase  of  the  project.  Within 
each  iteration,  the  technical  development  will  be  scheduled  around  a  monthly  sprint 
structure  that  forces  integration  and  evaluation  on  a  monthly  basis.  The  scope  of 
potential  ITS  developments  under  consideration  will  likely  exceed  the  schedule  and 
budget  available  to  develop  and  implement  a  prototype  ITS  within  the  IEDD  Operator 
Course.  The  work  plan  described  in  the  following  sections  describes  the  work  to  be 
performed  to  produce  an  ITS  prototype  to  support  evaluation  as  a  component  of  the 
scheduled  IEDD  courses.  Figure  3-15  provides  an  overview  of  the  inter-relationships 
between  tasks  associated  with  the  design,  implementation  and  evaluation  phases  of  the 
project.  The  remainder  of  this  section  will  describe  these  phases  in  more  detail. 
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DRDC  STATEMENT 
OF  WORK 


Figure  3-15:  Overview  of  the  inter-relationships  between  tasks  associated  with  the 
design,  implementation  and  evaluation  phases  of  the  project 


3.4.3  Task  Descriptions 

This  section  provides  a  list  of  recommendations  for  tasks  to  be  conducted  early  during 
the  next  phase  of  the  project  in  order  to  support  the  implementation  of  ITS  technologies 
within  the  IEDD  Operator  Course. 


3.4.3. 1  Questioning  Technique  Literature  Review 

In  order  to  create  new  instructional  material  as  part  of  the  ITS  scenario,  a  literature 
review  of  questioning  techniques  needs  to  be  undertaken.  The  review  should  explore 
questioning  and  interviewing  techniques  used  in  qualitative  psychology,  doctor-patient 
interviews,  police  investigations  (e.g.,  National  Investigation  Services).  The  review 
should  also  include  cognitive  biases  that  are  thought  to  affect  threat  assessment  and 
decision  making  (e.g.,  confirmation  bias,  cognitive  tunnelling),  and  strategies  by  which 
questioners  can  adopt  in  order  to  mitigate  them  (e.g.,  problem  solving  and  decision 
making  techniques  taught  as  part  of  aviation  Crew  Resource  Management).  The 
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intention  is  for  these  biases,  and  means  to  mitigate  them,  will  be  explored  within  the 
auspices  of  the  ITS  scenario.  The  review  should  also  include  any  lessons  learnt  from 
the  meeting  with  Maj  Terfry  at  the  School  of  Military  Intelligence  (CFB  Kingston) 
concerning  the  Tactical  Questioning  course,  and  Dr  Bruce  Spencer  at  the  Institute  for 
Information  Technology  (NRC,  Fredericton)  regarding  the  adaptive  learning  system  for 
doctor-patient  interviews.  Finally,  the  review  should  also  seek  to  identify  the  causes  of 
failure  of  IEDD  Operator  Course  students.  Correlates  to  be  examined  should  include 
existing  psychometric  data  collected  during  enrolment  into  the  CF.  This  work  should  be 
completed  in  collaboration  with  the  relevant  personnel  at  NDHQ  who  would  provide  the 
necessary  data. 


3.4.3.2  Develop  and  Validate  IEDD  ITS  Mission  Scenario 

The  second  task  is  for  the  human  factors  and  technical  team  to  work  together  to  flesh 
out  the  current  experimental  concept  into  a  detailed  storyboard  of  the  IEDD  ITS 
scenario.  The  storyboard  will  serve  as  the  requirements  specification  for  the  prototype. 
However,  an  agile  philosophy  is  recommended  -  the  purpose  of  the  storyboard  is  not  to 
unambiguously  specify  the  functionality  of  a  clearly  defined  system  -  rather,  the 
purpose  of  the  storyboard  is  to  clearly  communicate  the  functionality  of  the  desired 
prototype  system  between  the  scientific  authority,  and  the  CAE  PS  Human  Factors  and 
Modeling  and  Simulation  teams.  The  storyboard  should  define  more  functionality  than  is 
expected  for  the  prototype,  and  should  prioritize  functionality.  This  information  will  allow 
the  team  to  balance  scientific/experimental  significance  with  technical  complexity  to 
produce  the  most  effective  prototype  possible  given  the  fixed  schedule  and  budget 
constraints.  It  is  expected  that  the  storyboards  will  evolve  over  time  as  the  developing 
prototype  is  evaluated  and  insights  gained  are  fed  back  into  the  storyboard-based 
definition  of  the  system. 

The  storyboard  should  use  simple  artefacts  such  as  cue  cards,  PowerPoint  style 
graphics  and  simple  text  to  outline  the  script  for  the  IEDD  ITS  scenario  in  detail.  The 
storyboard  should  explore  the  option  paths.  The  storyboard  provides  the  development 
team  with  a  picture  of  the  system  that  needs  to  be  built,  and  makes  explicit  what  the 
technical  prototype  must  accomplish.  The  storyboard  also  provides  the  basis  for 
negotiation  and  refinement  within  the  team  as  strengths  and  limitations  of  technology 
are  encountered. 

An  initial  storyboard  will  be  developed  at  the  start  of  the  next  phase  of  the  ITS  project. 
The  storyboard  will  be  a  living  document  that  will  be  refined  and  modified  as  the 
prototype  is  iteratively  developed.  The  initial  storyboard  will  developed  during  a  scenario 
development  and  instructional  design  workshop.  This  workshop  will  support  the 
development  of  a  suitable  mission  scenario  for  the  implementation  of  ITS-related 
technologies  into  the  IEDD  Operator  Course.  This  should  be  held  in  May  2010  at  the 
School  of  Military  Engineering,  CFB  Gagetown  to  coincide  with  a  visit  to  the  Spring 
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2010  IEDD  Operator  Course.  This  workshop  will  also  develop  ITS  training  and 
evaluation  scenarios  based  on  the  mission  scenario.  Specifically,  Main  Teaching  Points 
(MTPs)  will  be  developed  for  the  following  Performance  Objectives  (POs)  from  the 
Qualification  Standard  (QS)  Improvised  Explosive  Device  (IEDD)  Operator  (reference: 
A-P3-002-IED/PH-H01):  PO  5.10  Good  Questioning  Technique,  and  PO  5.1 1  Task 
Appreciation  /  Threat  Assessment.  Finally,  evaluation  criteria  for  these  POs  will  also  be 
developed. 

Next,  the  data,  information  and  knowledge  required  to  support  the  experimental 
scenario  must  be  made  explicit.  An  initial  knowledge  representation  must  be  developed 
at  the  initiation  of  the  technical  work.  However,  this  task  is  expected  to  continue  in 
parallel  with  the  technical  development  over  the  course  of  the  development  of  the 
prototype.  Knowledge  must  be  represented  in  a  number  of  areas: 

1 .  The  central  knowledge  representation  is  the  ontology  of  IEDD  knowledge.  This 
ontology  must  classify  and  decompose  IED  devices.  It  must  define  the  render 
safe  procedures.  It  must  relate  the  classification  of  devices  to  the  render  safe 
procedures.  It  must  define  the  observations  of  the  environment  that  relate  to 
deducing  device  characteristics.  Finally,  it  must  make  explicit  the  inference  rules 
that  are  used  to  relate  the  observations  of  the  environment  to  the  deductions 
about  the  device  characteristics. 

2.  The  second  knowledge  representation  required  is  the  definition  of  the  question 
and  answer  structure.  Certain  of  the  observables  defined  in  the  IEDD  ontology 
are  mapped  to  answers  that  specific  NPC  characters  in  the  simulation  could 
provide  to  the  student.  The  question  and  answer  structure  must  make  explicit 
which  NPC’s  can  make  available  which  answers,  and  what  questions  are 
required  to  elicit  these  answers.  It  is  expected  that  this  structure  will  also  contain 
questions  and  answers  not  directly  related  to  the  IEDD  deductive  process,  as 
part  of  the  student’s  workload  is  the  separation  of  information  into  items  related 
to  IEDD,  and  items  note  related  to  IEDD. 

3.  The  third  knowledge  representation  required  is  the  mapping  of  the  remaining 
observations  to  items  discoverable  in  the  simulated  environment.  This  provides 
the  requirements  for  what  items  need  to  exist  in  the  simulated  environment  to 
support  the  IEDD  task.  Further,  it  is  expected  that  additional  items  will  be 
specified,  as  part  of  the  student’s  workload  is  the  separation  of  observations 
related  to  IEDD  from  observations  that  are  not  crucial  to  the  task  at  hand. 

4.  The  fourth  knowledge  representation  required  is  an  ontology  of  the  dynamic 
elements  required  to  support  the  experimental  scenario.  For  instance,  if  student 
or  NPC  actions  can  cause  IED  detonations,  or  if  student  actions  can  cause  NPC 
actions  such  as  departure  from  the  scene,  this  ontology  must  define  the  possible 
actions,  and  the  rules  by  which  these  actions  are  triggered. 
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The  technical  development  will  be  dependent  on  the  format  in  which  these  knowledge 
representations  will  be  captured.  The  initial  effort  of  the  CAE  PS  Human  Factors  and 
Modeling  and  Simulation  teams  will  be  the  definition  of  this  standard.  The  population  of 
these  knowledge  representations  can  then  proceed  in  parallel  with  the  technical 
development. 


3.4.3. 3  IEDD  ITS  Development:  Iteration  1 

The  purpose  of  the  first  iteration  is  to  produce  a  minimally  functional  implementation  of 
the  ITS  prototype  to  allow  early  evaluation  to  focus  priorities  for  the  second  iteration,  to 
assess  risk  in  areas  of  unknown  complexity,  and  to  allow  early  validation  of  the  system 
concept.  The  first  iteration  will  itself  be  composed  of  four  sprints  through  the  activities 
listed  in  the  following  sections.  Each  sprint  is  one  month  long  and  cycles  through 
definition,  design,  development,  integration  and  test.  This  iterative  structure  develops 
technology  in  a  fashion  that  quickly  develops  a  clear  picture  of  progress  and  risk.  This 
structure  enables  early  review  and  feedback  from  the  scientific  authority  and  the  SME 
community.  This  form  of  iterative  development  may  be  less  appropriate  for  the 
development  of  a  larger,  well  defined  system,  but  works  well  for  the  rapid  development 
of  a  prototype  for  experimentation. 

Iteration  1  will  be  composed  of  four  sprints,  each  composed  of  four  sequential  work 
tasks.  These  work  tasks  are  defined  in  the  following  sections.  The  early  sprints  will  be 
weighted  towards  the  specification  and  definition  activities.  The  later  sprints  will  be 
biased  towards  the  development,  integration  and  test  activities. 


3.4.3. 3.1  Sprint  #1  -1 :  Specify  Storyboards  and  Knowledge  Representation 

As  described  in  the  previous  sections  on  the  Storyboarding  and  Knowledge 
Representation  tasks,  the  technical  development  is  dependent  on  the  understanding  of 
the  system  requirements  as  detailed  by  the  storyboards,  and  on  the  nature  of  the 
supporting  data  in  the  form  of  the  knowledge  representations  that  will  support  the 
operation  of  the  system.  Each  sprint  will  select  a  focus  in  the  functionality  defined  in  the 
storyboard  and  use  this  focus  as  the  basis  for  the  development  effort  of  that  sprint. 


3.4.3. 3.2  Sprint  #1-2:  Define  System  Module  and  Interfaces 

Next,  the  functionality  that  is  the  focus  of  the  sprint  will  be  used  to  refine  the  system 
concept.  The  initial  system  concept  is  outlined  in  Section  3.2  of  this  report.  The  first 
stage  of  work  for  this  task  is  to  assign  the  functionality  to  the  modules  defined  in  the 
system  concept.  Next,  the  interfaces  between  the  modules  will  be  defined/refined  based 
on  the  allocation  of  functionality.  This  will  allow  the  development  of  the  modules  to 
proceed  in  a  more  loosely  coupled  fashion. 
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3.4.3. 3.3  Sprint  #1-3:  Develop  Modules 

Next,  the  functionality  assigned  to  each  module  must  be  implemented.  Early  sprints  will 
focus  on  the  selection  of  supporting  COTS/GOTS  technology  and  tools  to  provide  as 
much  of  the  required  functionality  as  possible.  Later  sprints  will  focus  on  the  extension 
of  these  tools  to  provide  required  functionality,  or  the  development  of  new  functionality 
and  interface  implementations  where  required. 


3.4.3. 3.4  Sprint  #1-4:  Integrate  and  Test 

The  final  stage  of  each  sprint  is  the  integration  of  the  new  developments  into  a 
functioning  system  and  the  testing  of  this  system.  Typically  a  level  of  integration  is 
pushed  during  the  development  phase,  but  the  final  integration  and  test  activity  in  each 
sprint  provides  a  more  structured  activity  to: 

1 .  Verify  that  the  system  concept  provides  the  required  functionality; 

2.  Verify  that  the  selected  tools  and  technologies  operate  as  expected,  and 

3.  Allow  informal  review  of  the  developing  system  by  the  scientific  authority  and  by 
SME’s  to  provide  early  validation  that  the  system  concept  and  storyboards  are  a 
good  representation  of  the  end  user  needs. 

The  Integration  and  Test  phase  of  the  last  sprint  of  the  first  iteration  will  be  a  more 
formal  evaluation  of  the  system.  If  SME  availability  is  limited,  SME  involvement  should 
be  focused  on  this  integration  and  test  phase. 


3.4.3.4  IEDD  ITS  Development:  Iteration  2 

At  the  initiation  of  the  second  iteration,  the  first  iteration  has  produced  a  minimally 
functional  prototype  that  has  been  tested  and  reviewed.  At  this  point,  the  system 
storyboards  and  knowledge  representations  will  have  been  refined,  and  the  detailed 
experimental  plan  should  be  available.  The  initiation  of  the  second  iteration  provides  an 
opportunity  for  the  scientific  authority  and  for  the  project  team  to  review  the  progress  to 
date,  the  functionality  of  the  initial  prototype,  and  the  utility  of  the  selected  tools  and 
techniques  for  the  project  objectives.  This  review  will  provide  an  opportunity  to  refine  the 
project  objectives  and  provide  clarity  to  the  technical  priorities  for  iteration  2. 

The  second  iteration  will  follow  the  pattern  of  the  first  iteration.  The  second  iteration  will 
be  composed  of  four  sprints,  each  one  month  in  length.  The  objective  of  the  second 
iteration  is  to  develop  the  ITS  prototype  that  will  support  the  experimental  plan.  The  first 
three  sprints  are  identical  to  the  first  iteration;  whereas  the  scope  of  the  fourth  sprint 
(Evaluation)  is  described  in  more  detail  below. 
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3.4.3.4.1  Sprint  #2-4:  Evaluate  Prototype 

The  Test  and  Evaluation  phase  of  the  last  sprint  of  iteration  2  will  be  a  formal  evaluation 
of  the  ITS  prototype.  This  final  evaluation  will  allow  the  scientific  authority  to  verify  that 
the  prototype  supports  the  evaluation  plan,  and  to  validate  that  the  developed 
functionality  addresses  the  objectives  of  the  ITS  project.  The  outcome  of  this  evaluation 
is  a  clear  picture  of  the  fitness  of  the  prototype  to  support  the  project’s  evaluation 
objectives.  The  evaluation  plan  for  the  IEDD  ITS  is  described  in  more  detail  in  Section  4 
of  this  report. 


3.4.3. 5  Technology  Watch 

Given  the  rapid  pace  at  which  gaming,  and  modeling  and  simulation-based  technologies 
are  evolving,  it  is  important  that  the  ITS  project  is  aware  of  the  current  state-of-the-art  of 
both  modeling  and  simulation  technologies,  and  adaptive  learning  and  intelligent 
tutoring  technologies.  In  addition,  it  is  also  important  for  the  ITS  project  to  maintain  and 
strengthen  the  relationships  with  the  stakeholders  established  during  the  first  phase  of 
the  project.  An  ongoing  task  during  the  next  phase  of  the  project  will  therefore  be  a 
Technology  Watch. 


3.4.4  Provisional  Schedule  for  IEDD  ITS  Implementation  and  Evaluation 

The  following  section  describes  a  provisional  schedule  for  the  implementation  and 
evaluation  of  the  IEDD  Operator  Course  ITS.  The  schedule  is  driven  by  a  number  of 
factors.  Specifically: 

•  The  project  will  be  completed  by  31st  March  2012. 

•  The  evaluation  of  the  IEDD  ITS  will  be  scheduled  to  coincide  with  the  running  of 
the  IEDD  Operator  Course  at  CFB  Gagetown.  The  course  is  run  three  times  per 
year;  generally  the  spring,  summer  and  fall.  In  order  to  capitalise  on  these 
opportunities  the  following  evaluation  activities  are  scheduled5: 

o  Mission  scenario  workshop.  Development  and  validation  of  IEDD  ITS 
mission  scenario  and  taxonomic  analysis.  To  coincide  with  the  spring 
2010  course.  Further  opportunities  to  refine  the  scenario  and  analysis  will 
coincide  with  the  summer  and  fall  2010  courses. 

o  Baseline  data  collection.  Administration  of  questionnaires  and  collection  of 
baseline  psychophysiological  data  from  course  students  will  coincide  with 


5 


The  evaluation  plan  for  the  IEDD  ITS  is  described  in  more  detail  in  Section  4  of  this  report. 
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the  spring  2010  and  spring  201 1  courses.  This  provides  a  total  of 
approximately  40  participants  (20  per  course). 

o  Prototype  evaluation.  A  dry-run  with  course  students  and  instructors  in 
order  to  get  feedback  regarding  the  usability  of  the  IEDD  ITS  and  the 
quality  of  instructional  feedback.  The  prototype  evaluation  will  coincide 
with  the  spring  201 1  course. 

o  ITS  evaluation  sessions.  The  formal  evaluation  of  the  IEDD  ITS  with 
course  students  will  coincide  with  the  summer  and  fall  201 1  courses.  This 
provides  a  total  of  approximately  40  participants  (20  per  course). 

Figure  3-16  presents  the  provisional  implementation  and  evaluation  schedule  for  the 
IEDD  ITS.  The  schedule  is  driven,  in  part,  by  the  six  IEDD  Operator  Courses  to  be  run 
in  2010  and  201 1 .  The  IEDD  development  and  evaluation  spirals,  as  well  as  the  data 
collection  activities,  have  been  scheduled  to  coincide  with  these  courses. 
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Figure  3-16:  Provisional  IEDD  ITS  implementation  and  evaluation  schedule 
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3.5  Recommended  Implementation  Plan  for  the  IEDD  Mobile 
Learning  ITS 

Parallel  activities  were  undertaken  to  review  and  identify  appropriate  mobile  devices  for 
adaptive  learning  and  intelligent  tutoring  technologies  (Kramer,  Tryan  and  Banbury, 
2010),  with  the  intention  of  exploring  the  feasibility  of  porting  content  from  the  IEDD 
Operator  Course  ITS  on  to  the  mobile  device. 

At  a  minimum,  the  CAE  PS  development  team  will  exploit  any  lessons  learnt  from  the 
development  of  the  ‘PC-based’  IEDD  ITS  to  recommend  how  adaptive  learning  and 
intelligent  tutoring  technologies  can  be  successfully  ported  across  to  the  mobile 
platform.  Ideally,  the  CAE  PS  team  will  develop  a  prototype  version  of  the  IEDD  ITS  on 
a  mobile  device;  however,  this  will  depend  upon  the  development  schedule  of  the  ‘PC- 
based’  version  of  the  IEDD  ITS. 

The  evaluation  of  adaptive  learning  and  intelligent  tutoring  technologies  on  a  mobile 
learning  platform  will  be  limited  to  an  informal  evaluation  by  ITS  project  team  personnel, 
at  a  minimum.  Ideally,  IEDD  course  students  would  also  take  part  in  the  evaluation 
process;  however,  their  involvement  will  depend  upon  the  development  schedule  of  the 
‘PC-based’  version  of  the  IEDD  ITS.  There  are  no  plans  to  formally  evaluate  (i.e. , 
conduct  rigorous  experimentation)  the  mobile  version  of  the  IEDD  ITS  under  the 
auspices  of  the  current  project. 
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4  EVALUATION  PLAN 

This  section  of  the  report  describes  the  evaluation  plan  for  the  IEDD  Operator  Course 
ITS.  The  first  section  describes  earlier  efforts  by  Banbury,  Roberts,  Hartlen  and  Unrau 
(2009)  to  identify  the  objectives,  constraints,  and  generic  experimental  design  for 
evaluating  a  hypothetical  example  of  the  IEDD  ITS.  The  second  section  describes  the 
methodology  and  approximate  schedule  for  activities  pertaining  to  the  evaluation  of  the 
IEDD  ITS. 


4.1  Introduction 

Banbury,  Roberts,  Hartlen  and  Unrau  (2009)  presented  an  overview  of  issues  pertaining 
to  the  evaluation  of  the  implementation  of  the  adaptive  learning  technologies  within  the 
IEDD  course.  These  issues  relate  to  the  examination  of  the  pedagogical  impact  of  the 
adaptive  learning  enhancements  on  the  IEDD  course  within  the  context  of  both  self- 
paced  individual  learning  and  instructor-led  group  learning  situations.  Their  report 
provides  information  with  regards  to  the  methodological  approach  suggested  for  the 
evaluation.  It  also  comprises  a  discussion  of  the  participants,  experimental  design,  and 
measures  of  effectiveness,  and  a  number  of  issues  for  the  development  of  the 
evaluation  methodology.  Banbury  et  al.’s  report  represents  the  first  step  in  the 
development  of  the  IEDD  ITS  evaluation  methodology  and  the  topics  discussed  within  it 
were  based  on  a  number  of  assumptions  and  constraints  that  had  been  identified  at  that 
time. 

This  section,  therefore,  provides  an  update  to  the  evaluation  plan  presented  by  Banbury 
and  colleagues.  To  avoid  undue  repetition  in  this  report,  the  reader  is  advised  to  refer  to 
both  evaluation  documents.  The  evaluation  plan  and  schedule  described  in  the  following 
section  is  driven  by  a  number  of  constraints.  Specifically: 

•  The  project  will  be  completed  by  31st  March  2011. 

•  Baseline  Data  Collection  to  coincide  with  the  spring  2010  and  spring  201 1  IEDD 
Operator  Courses.  This  provides  a  total  of  approximately  40  participants  (20  per 
course). 

•  IEDD  ITS  Evaluation  to  coincide  with  the  summer  and  fall  201 1  IEDD  Operator 
Courses.  This  provides  a  total  of  approximately  40  participants  (20  per  course). 

Both  the  Baseline  Data  Collection  and  IEDD  ITS  Evaluation  phases  are  described  in 
more  detail  below.  It  is  important  to  note  that  the  methodology  is  likely  to  evolve  as 
further  assumptions  and  constraints  are  identified.  As  such,  this  section  does  not 
represent  the  final  methodology  that  will  be  used  in  the  evaluations,  but  is  instead 
intended  to  stimulate  debate  and  discussion. 
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Finally,  as  discussed  in  section  2.2.1 .2,  instructors  from  the  IEDD  operator  course 
stated  that  no  control  condition  in  which  some  students  have  a  'lesser  learning 
experience'  will  be  acceptable.  As  such,  the  direct  evaluation  of  physiological  monitoring 
technologies  was  not  possible  given  that  these  likely  performance  enhancing 
technologies  could  not  be  ‘turned  off  in  a  control  condition. 


4.2  Phase  I:  Baseline  Data  Collection 

In  order  to  evaluate  the  impact  of  adaptive  learning  and  intelligent  tutoring  technologies 
on  the  quality  of  the  questioning  technique  and  threat  assessment  instruction,  it  is 
important  to  collect  baseline  data  prior  to  the  implementation  of  the  IEDD  ITS  within  the 
course. 

The  first  phase  of  the  evaluation,  therefore,  comprises  Baseline  Data  Collection 
activities  to  be  undertaken  during  the  spring  2010  and  spring  201 1  IEDD  Operator 
Courses.  The  baseline  data  collection  comprises  the  following  activities: 

•  Overall  Course  Performance  Assessment; 

•  Index  of  Learning  Styles  (ILS)  Questionnaire;  and, 

•  Conscientiousness  Personality  Trait  Questionnaire. 

These  data  collection  activities  are  described  in  more  detail  in  the  following  section. 


4.2.1  Overall  Course  Performance  Assessment 

A  quick  and  effective  way  of  assessing  student  performance  is  to  use  a  direct 
observation  methodology  known  as  behavioural  markers.  Behavioural  markers  refer  to 
a  prescribed  set  of  behaviours  indicative  of  some  aspect  of  performance.  Typical 
behaviours  are  listed  in  relation  to  component  skills.  Some  marker  systems  have  been 
designed  to  rate  operators’  behaviours  in  response  to  predetermined  scenario  events. 
Typically,  researchers  produce  a  list  of  acceptable  behavioural  responses  to  generated 
events  or  tasks  (e.g.,  mission  analysis,  adaptability-flexibility,  decision-making,  SA,  and 
communication).  Behaviours  are  linked  specifically  to  stimulus  events  in  a  scenario  (i.e., 
these  are  predefined  into  a  set  of  acceptable  behaviours  or  task  responses)  and  then 
they  are  rated  as  present  or  absent.  The  ratings  are  normally  conducted  using 
personnel  experienced  in  the  domain  (e.g.,  a  course  instructor). 
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TDE  OPS  TASK  CONDUCT 

Ref 

Arrival  Actions  @  RV 

Yes 

Y/B 

No 

Ref 

Evidence  marking 

Yes 

Y/B 

No 

Brief  from  OSC/TET 

What  is  the  Forensics  required?  (Critical) 
Where  is  the  evidence  (Critical) 

Site  Integrity 

Hazards  Explosive 

Bio  Hazard  protocols 

HAZMAT  Protocols 

Time  on  Target  needed 

Determined  Tactical  considerations 


Ref 

Questioning  Witnesses 

Yes 

Y/B 

No 

Good  questioning  technique 

Task  Appreciation  /  TA  for  Hazards 

Ref 

Briefing  &  Discussion 

Yes 

Y/B 

No 

Briefed  task  pers 

Advised  OSC  &  TET  appropriately 

Timely  passage  of  info 

Maintains  Command  &  Control  of  task 

Devised  safe,  efficient  plan 

Forensic  Protocols  adhered  to  by 
tasked/breifed  Pers 

Concurrent  Activity 

Flexible  for  TOT 

Prioritized  Evidence  collection 

Collected  relevant  evidence 

Who 

What 

When 

Where 

Control  Number 


Ref 

Forensic  Philosophy  Followed 

Yes 

Y/B 

No 

Search  Teq  proper  for  TOT 

Changed  gloves  as  req  for  damage 

Changed  gloves  as  req  for  cross  contamination 

Correct  container  used  when  shipping 

Munitions  Items  IDd  correctly 

Preserve  &  Collect  Relevant  Forensic 
Evidence 

Y/B 

No 

ICP  Safety 

Cordon  Safety 

Protective  Equipment 

HAZMAT 

BIO 

Explosives  Safety 

Weapon  /  equipment  safety 

Revert  back  to  IED  Tech  if  secondary 
Explosive  Hazard  found 

Ref 

Crater/Debris  Field  Exploitation 

Yes 

Y/B 

No 

Effective  equipment  selected 

Effective  equipment  employment 

Took  and  Noted  relevant  Measurements 
and  locations 

Relevant  Pics  taken 

Pics  taken  in  organized  sequence 

Sufficient  amount  and  angles  of  pics 

Re-planned,  re-briefed? 

Gathered  sufficient  measurements 

Ref 

Evidence  Collection 

Yes 

Y/B 

No 

Effective  equipment  selected 

Effective  equipment  employment 

Glove  protocol 

Collection  done  forensically  sound 

Re-planned,  re-briefed? 

Evidence  locations  noted 

Biometric  collection 

Evidence  mixing/contamination 
consideration 

Proper  sample  sizes  taken 

Figure  4-1 :  Example  of  a  behavioural  marker  checklist  for  the  IEDD  Operator  Course 
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An  example  of  a  behavioural  marker  checklist  is  presented  in  Figure  4-1 .  The  checklist 
is  currently  used  to  assess  student  performance  on  the  existing  IEDD  Operator  Course 
and  is  based  on  a  three-point  scoring  regimen  (i.e. ,  yes  /  yes  but  /  no).  Alternatively,  all 
behavioural  marker  elements  can  be  rated  on  a  4-point  Likert  scale  ranging  from,  for 
example,  1  (poor),  2  (minimum  expectations),  3  (standard),  to  4  (outstanding).  The  use 
of  a  Likert  scale  provides  interval-ratio  data  which  allow  the  use  of  inferential  statistics, 
such  as  regression  and  correlations.  The  development  of  the  Likert  scale  for  the  student 
assessments  will  be  undertaken  with  course  instructors  during  the  mission  scenario 
workshop  to  be  held  during  the  spring  2010  course.  The  workshop  will  explore  the 
means  by  which  the  existing  evaluation  checklist  can  be  further  decomposed  to  capture 
the  nuances  of  the  questioning  skills  underlying  threat  assessment.  For  example,  the 
present  evaluation  checklist  has  only  two  items  pertaining  to  QT  and  threat  assessment. 

In  addition,  the  following  data  will  also  be  collected: 

•  Overall  course  pass  /  failure  rate;  and, 

•  Student  demographic  information  such  as:  time  in  OF,  rank,  trade  prior  to  course, 
and  relevant  psychometric  data  collected  by  the  OF  during  the  student’s 
application  and  enrolment  process. 

The  analysis  of  the  performance  assessments  will  comprise  the  following  activity: 

1 .  Statistical  correlation  of  student  course  performance  assessment  scores  with 
student  demographic  information.  This  analysis  will  determine  if  a  particular 
demographic  variables  increases  (or  reduces)  the  learning  performance  of  the 
course  students. 

4.2.2  Index  of  Learning  Styles  (ILS)  Questionnaire 

A  baseline  of  learning  styles  should  be  assessed  using  the  Felder  Solomon  Learning 
Styles  Index  online  questionnaire  administered  to  capture  the  initial  values  used  to 
represent  the  learner’s  style  and  to  determine  what  content  needs  to  be  developed 
within  the  IEDD  ITS  in  order  to  support  the  learning  styles  of  the  course  students.  In 
addition,  baseline  learning  styles  data  should  be  compared  against  the  style  of  current 
IEDD  course  teaching  styles  to  see  if  the  areas  where  students  are  failing  are  indeed 
those  which  show  a  mismatch  between  learning  and  teaching  style,  and/or  presentation 
of  information.  Finally,  as  suggested  as  in  Tripp  and  Moore  (2007),  that  IEDD  Operator 
Course  instructors  themselves  should  also  be  assessed  with  an  index  of  learning  styles, 
and  to  compare  their  own  styles  with  those  of  their  students.  The  goal  is  that  teaching 
can  be  improved  when  teaching  styles  are  better  matched  with  learning  styles. 
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The  Index  of  Learning  Styles  (ILS)  developed  by  Richard  M.  Felder  and  colleagues 
comprises  questions  related  to  four  dimensions:  Active/Reflective,  Sensitive/Intuitive, 
Sequential/Global  and  Visual/Verbal.  These  dimensions  are  described  as  follows: 

•  Active  processing  (prefers  active  student  participation  in  groups)  or  reflective 
processing  (prefers  passive  student  participation  by  themselves  or  with  one 
familiar  partner); 

•  Sensing  perception  (prefers  concrete,  practical  content)  or  intuitive  perception 
(prefers  abstract,  conceptual  content); 

•  Visual  input  (prefers  visual  presentation)  or  verbal  input  (prefers  written  and 
spoken  presentation);  and, 

•  Sequential  understanding  (prefer  linear  thinking)  or  global  understanding  (prefer 
holistic  thinking). 

Each  dimension  consists  of  two  categories,  and  each  category  has  a  score  ranging  from 
1  to  1 1 .  Scores  ranging  from  1  to  3  indicate  mild  or  well  balanced  between  the  two 
categories.  For  scores  between  5  and  7,  a  moderate  preference  is  indicated,  which 
means  favouritism  for  one  of  the  two  categories.  Scores  between  9  and  1 1  indicate  a 
very  strong  preference,  meaning  difficulty  with  learning  where  the  environment  does  not 
support  that  category.  The  ILS  can  be  administered  on  line  and  the  results  are  emailed 
back  to  the  participant  automatically  (e.g.,  http://www.enqr.ncsu.edu/learninqstyles/ 
ilsweb.html). 

The  analysis  of  the  ILS  questionnaire  results  will  comprise  the  following  activities: 

1 .  Statistical  correlation  of  ILS  scores  with  course  performance  outcome  measures. 
This  analysis  will  determine  if  a  particular  learning  style  promotes  (or  reduces) 
the  learning  performance  of  the  course  students. 

2.  Determination  of  ILS  dimensions  clusters  across  all  students  on  the  course.  This 
assessment  will  be  used  to  determine  what  content  needs  to  be  developed  within 
the  IEDD  ITS  in  order  to  support  the  learning  styles  of  the  course  students.  This 
assessment  will  be  performed  subjectively  by  the  CAE  PS  team. 

3.  Determination  of  the  compatibility  between  the  learning  styles  of  the  student,  and 
the  instructional  style  of  the  instructor  and  instructional  content  of  the  course 
(e.g.,  predominately  text-based  materials  and  so  on).  This  assessment  will  be 
used  to  determine  the  effectiveness  of  current  approaches  to  teaching  on  the 
IEDD  Operator  Course.  This  assessment  will  be  performed  subjectively  by  the 
CAE  PS  team. 
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4.2.3  Conscientiousness  Personality  Trait  Questionnaire 

Durso  and  Dattel  (2004)  suggest  that  differences  in  the  ability  to  maintain  Situation 
Awareness  during  an  air  traffic  control  task  are  related  to  differences  in  personality; 
specifically  the  personality  trait  of  conscientiousness.  Conscientiousness  is  the  trait  of 
being  painstaking  and  careful,  and  includes  the  tendency  to  show  self-discipline,  act 
dutifully,  and  aim  for  achievement.  The  trait  shows  a  preference  for  planned  rather  than 
spontaneous  behaviour,  carefulness,  thoroughness,  organization,  deliberation  (the 
tendency  to  think  carefully  before  acting),  and  need  for  achievement. 

Conscientiousness  is  one  of  five  super-ordinate  traits  in  the  “Big  Five  model”  of 
personality  which  also  consists  of  extraversion,  neuroticism,  openness  to  experience, 
and  agreeableness  (Costa  and  McCrae,  1992).  Discussions  with  representatives  of  the 
IEDD  Operator  Course  suggested  that  students  who  had  previously  been  Navy  Divers 
were  often  the  strongest  students  on  the  course.  Given  that,  from  an  anecdotal  point  of 
view,  professional  divers  need  to  be  conscientious,  it  is  not  surprising  that  students  with 
a  background  in  professional  diving  would  score  highly  on  a  personality  trait  of 
conscientiousness,  and  in  doing  so,  show  good  Situation  Awareness  acquisition  skills 
(i.e.,  threat  assessment)  during  the  IEDD  Operator  Course. 

In  summary,  given  the  cognitive  similarities  between  the  general  phenomenon  of 
Situation  Awareness,  and  the  situation  /  threat  assessment  activities  undertaken  by 
IEDD  operators,  we  would  expect  that  IEDD  Operator  Course  students  that  score  highly 
on  measures  of  ‘consciousness’  should  also  do  well  on  the  course. 

The  Big  Five  personality  questionnaire,  which  includes  the  assessment  of  the  sub¬ 
ordinate  trait  of  conscientiousness,  can  be  administered  on-line  and  the  results  are 
emailed  back  to  the  participant  automatically  (e.g.,  http://www.outofservice.com/ 
biqfive/). 

The  analysis  of  the  Big  Five  questionnaire  results  will  comprise  the  following  activity: 

1 .  Statistical  correlation  of  conscientiousness  trait  scores  with  course  performance 
outcome  measures.  This  analysis  will  determine  if  this  dimension  of  personality 
promotes  (or  reduces)  the  learning  effectiveness  of  the  course  students. 

4.3  Phase  II:  ITS  Evaluation  Sessions 

In  order  to  complete  the  evaluation  of  IEDD  ITS  on  the  quality  of  the  questioning 
technique  and  threat  assessment  instruction,  the  second  phase  of  the  evaluation  is  to 
collect  data  following  the  implementation  of  the  IEDD  ITS  within  the  course. 
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The  second  phase  of  the  evaluation,  therefore,  comprises  ITS  evaluation  sessions  to  be 
undertaken  during  the  summer  201 1  and  fall  201 1  IEDD  Operator  Courses.  The  ITS 
evaluation  sessions  comprise  the  following  activities: 

•  Overall  Course  Performance  Assessment; 

•  Index  of  Learning  Styles  (ILS)  Questionnaire; 

•  Conscientiousness  Personality  Trait  Questionnaire; 

•  IEDD  ITS  Usability  Questionnaire;  and, 

•  Situation  Awareness  and  Workload  questionnaires. 

These  data  collection  activities  are  described  in  more  detail  in  the  following  section. 

4.3.1  Overall  Course  Performance  Assessment 

The  evaluation  checklist  of  behavioural  markers  developed  for  the  baseline  data 
collection  phase  will  be  re-administered.  Once  again,  the  ratings  will  be  conducted  using 
personnel  experienced  in  the  domain  (e.g.,  instructor).  In  addition,  the  CAE  PS  team  will 
conduct  de-briefing  sessions  with  the  raters  to  focus  on  their  evaluation  of  the 
questioning  skills  underlying  threat  assessment.  This  qualitative  feedback  often 
produces  key  insights  with  which  to  interpret  the  quantitatative  data  collected  using  the 
evaluation  checklists. 

In  addition,  the  following  data  will  also  be  collected: 

•  Overall  course  pass  /  failure  rate;  and, 

•  Student  demographic  information  such  as:  time  in  CF,  rank,  trade  prior  to  course, 
and  relevant  psychometric  data  collected  by  the  CF  during  the  student’s 
application  and  enrolment  process. 

The  analysis  of  the  ILS  questionnaire  results  will  comprise  the  following  activities: 

1 .  Statistical  comparison  between  course  performance  assessment  scores 
collected  during  the  baseline  sessions  and  those  collected  during  the  IED  ITS 
evaluation  sessions.  This  comparison  will  determine  the  pedagogical  impact  of 
the  IEDD  ITS;  in  other  words,  the  effectiveness  of  the  adaptive  learning  and 
intelligent  tutoring  technologies  to  improve  the  learning  experience  of  the  course 
students. 
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2.  Statistical  correlation  of  student  course  performance  assessment  scores  with 
student  demographic  information.  This  analysis  will  determine  if  a  particular 
demographic  variables  increases  (or  reduces)  the  learning  performance  of  the 
course  students. 

3.  Statistical  comparison  of  student  demographic  information.  This  analysis  will 
determine  if  a  particular  demographic  variables  increases  (or  reduces)  the 
learning  performance  of  the  course  students. 

4.  Determination  of  the  consistency  of  the  demographic  profile  between  students  in 
the  baseline  data  collection  sessions  and  students  in  the  ITS  evaluation 
sessions.  This  assessment  will  be  performed  using  inferential  statistics  whenever 
possible.  The  intention  of  this  analysis  is  to  determine  whether  the  demographic 
composition  of  the  students  across  the  baseline  and  evaluation  sessions  remains 
relatively  stable.  If  so,  any  statistical  differences  observed  between  the  baseline 
and  evaluation  sessions  (point  1  above)  cannot  be  accounted  for  by  these 
(confounding)  variables. 

4.3.2  Index  of  Learning  Styles  (ILS)  Questionnaire 

The  Index  of  Learning  Styles  (ILS)  will  be  re-administered  to  students  in  the  ITS 
evaluation  sessions.  The  analysis  of  the  ILS  questionnaire  results  will  comprise  the 
following  activities: 

1 .  Statistical  correlation  of  ILS  scores  with  course  performance  outcome  measures. 
This  analysis  will  determine  if  a  particular  learning  style  promotes  (or  reduces) 
the  learning  performance  of  the  course  students.  We  would  expect  this  effect  (if 
any)  to  weaken  given  that  the  IEDD  ITS  provides  instruction  consistent  with  the 
students’  learning  style. 

2.  Determination  of  ILS  dimensions  clusters  across  all  students  on  the  course.  This 
assessment  will  be  used  to  determine  if  the  content  developed  for  the  IEDD  ITS 
from  questionnaire  data  collected  during  the  baseline  data  collection  sessions 
matches  the  learning  styles  of  the  students  in  the  ITS  evaluation  sessions.  This 
assessment  will  be  performed  using  inferential  statistics  whenever  possible.  The 
intention  of  this  analysis  is  to  determine  whether  the  learning  styles  of  the 
students  across  the  baseline  and  evaluation  sessions  remains  relatively  stable.  If 
so,  any  statistical  differences  observed  between  student  learning  performance  in 
the  baseline  and  evaluation  sessions  cannot  be  accounted  for  by  this 
(confounding)  variable. 
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4.3.3  Conscientiousness  Personality  Trait  Questionnaire 

The  Big  Five  personality  questionnaire  will  be  re-administered  to  students  in  the  ITS 
evaluation  sessions.  The  analysis  of  the  Big  Five  questionnaire  results  will  comprise  the 
following  activities: 

The  analysis  of  the  Big  Five  questionnaire  results  will  comprise  the  following  activities: 

1 .  Statistical  correlation  of  conscientiousness  trait  scores  with  course  performance 
outcome  measures.  This  analysis  will  determine  if  this  dimension  of  personality 
promotes  (or  reduces)  the  learning  performance  of  the  course  students.  We 
would  expect  this  effect  (if  any)  to  weaken  given  the  IEDD  ITS  explicitly 
encourages  a  rigorous  and  logical  style  of  threat  assessment  by  explicitly 
representing  the  product  of  threat  assessment  in  real-time  using  the  SMAP. 

2.  Determination  of  the  consistency  of  the  conscientiousness  trait  scores  between 
students  in  the  baseline  data  collection  sessions  and  students  in  the  ITS 
evaluation  sessions.  This  assessment  will  be  performed  using  inferential 
statistics  whenever  possible.  The  intention  of  this  analysis  is  to  determine 
whether  the  personality  profile  of  the  students  across  the  baseline  and  evaluation 
sessions  remains  relatively  stable.  If  so,  any  statistical  differences  observed 
between  student  learning  performance  in  the  baseline  and  evaluation  sessions 
cannot  be  accounted  for  by  this  (confounding)  variable. 

4.3.4  IEDD  ITS  Usability  Questionnaire 

The  Technology  Acceptance  Model  (TAM)  is  an  information  systems  theory  that  models 
how  operators  come  to  accept  and  use  a  technology  (Davis,  1989).  The  model  suggests 
that  when  operators  are  presented  with  a  system,  a  number  of  factors  will  influence  their 
decision  about  how  and  when  they  will  use  it,  notably: 

•  Perceived  Usefulness.  This  is  defined  as  ‘the  degree  to  which  a  person  believes 
that  using  a  particular  system  would  enhance  his  or  her  job  performance’. 

•  Perceived  Ease-of-Use.  This  is  defined  as  ‘the  degree  to  which  a  person  believes 
that  using  a  particular  system  would  be  free  from  effort.’ 

The  TAM  utilizes  a  questionnaire  that  has  been  assessed  for  robustness  across 
populations  and  predictive  validity.  Studies  have  found  high  reliability  and  good  test- 
retest  reliability  and  have  found  that  the  instrument  had  predictive  validity  for  intent  to 
use,  self-reported  usage  and  attitude  toward  use.  The  sum  of  this  research  has 
confirmed  the  validity  of  the  instrument,  and  supports  its  use  with  different  populations 
of  operators  and  different  software  choices. 
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This  questionnaire  will  be  adapted  in  order  to  test  the  high-level  usability  aspects  of  the 
IEDD  ITS  GUI  (e.g.,  suitability  of  screen  windows,  interaction  devices  and  so  on)  to 
assess  student  perceptions  of  system  ease  of  use,  and  the  perceived  utility  of  the  IEDD 
ITS  functionality  and  capabilities  (e.g.,  intelligent  tutor,  SMAP  and  so  on)  to  assess 
student  perceptions  of  system  usefulness.  Ratings  will  be  based  on  a  five-point  Likert 
scale;  ranging  from  1 :  Strongly  Disagree  to  5:  Strongly  agree.  The  specific  content  of 
this  questionnaire  will  be  developed  once  the  IEDD  ITS  has  reached  a  mature  stage  in 
its  development  and  its  functionality  and  capabilities  are  clearly  defined. 


4.3.5  Situation  Awareness  and  Workload  questionnaires 

Situation  Awareness  (SA)  is  a  key  determinant  of  I  ED  disposal  activities  and  relates  to 
the  ability  of  the  IEDD  operator  to  maintain  awareness  of  task-relevant  entities  in  their 
immediate  environment.  The  measurement  of  SA  within  the  context  of  the  IEDD  ITS 
evaluation  is  important  given  the  importance  of  the  appropriate  questioning  of 
witnesses,  and  the  subsequent  threat  assessment  that  is  derived  from  it.  A  validated 
measure  of  a  student’s  SA  following  the  threat  assessment  process  will  help  to  verify 
any  conclusions  we  make  about  the  effectiveness  of  the  SMAP  tool;  essentially,  an 
explicit  representation  of  their  SA.  There  are  several  well-validated  measures  of  SA  that 
are  readily  available  and  applicable  to  these  types  of  validation.  However,  the  decision 
over  which  measure  to  use  will  be  taken  once  the  IEDD  ITS  has  reached  a  mature 
stage  in  its  development  and  its  functionality  and  capabilities  are  clearly  defined. 

Another  key  determinant  of  task  performance  is  the  workload  experienced  by  the 
student  when  conducting  their  threat  assessment  within  the  scenario.  Workload  can  be 
conceptualised  in  terms  of  both  physical  and  mental  effort.  Workload  data  provides  a 
useful  insight  into  the  task  difficulty  experienced  by  the  student.  This  insight  allows  other 
conclusions  about  the  student’s  learning  experienced  to  be  interpreted  in  this  light. 

Once  again,  there  are  several  well-validated  measures  of  workload  that  are  readily 
available  and  applicable  to  these  types  of  validation.  However,  the  decision  over  which 
measure  to  use  will  be  taken  once  the  IEDD  ITS  has  reached  a  mature  stage  in  its 
development  and  its  functionality  and  capabilities  are  clearly  defined. 
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5  CONCLUSIONS 

The  final  section  of  the  report  summarises  the  results  of  the  stakeholder  analysis,  and 
provides  recommendations  for  the  next  steps  for  both  the  implementation  plan  and 
evaluation  plan. 

5.1  Summary  of  Results 

The  results  of  the  stakeholder  analysis  demonstrate  a  range  of  maturing  technologies 
that  can  be  usefully  applied  to  the  development  of  an  intelligent  tutoring  system 
designed  to  augment  the  IEDD  Operator  Course,  as  part  of  the  CF  e-learning  initiative. 
The  recommendations  from  the  stakeholder  meetings,  and  subsequent  analysis  of  the 
stakeholders  themselves,  are  described  below. 


5.1.1  Stakeholder  Meetings 

In  summary,  the  stakeholder  meetings  were  a  success  insofar  as  they  provided  a  rich 
source  of  information  and  advice  for  the  implementation  of  ITS  technologies,  and  an 
opportunity  to  establish  relationships  with  the  stakeholders  of  the  ITS  project.  The  key 
outcomes  were  as  follows: 

1 .  The  selection  of  the  IEDD  Operator  Course  for  implementation  of  adaptive 
learning  and  intelligent  tutoring  projects  is  appropriate. 

2.  The  IEDD  Operator  Course  personnel  are  keen  to  support  the  ITS  project  in 
terms  of  access  to  course  materials,  instructors  and  students. 

3.  In  terms  of  the  selection  of  the  appropriate  simulation  and  modeling  tools  to 
implement  the  IEDD  Operator  Course  ITS,  the  clear  consensus  was  that  Second 
Life  was  not  an  appropriate  environment  (as  we  had  first  thought),  and  that 
Bohemia  Interactive’s  VBS2  should  be  the  tool  of  choice. 

4.  DLSE  has  offered  to  provide  VBS2  support  to  DRDC  (and  CAE  PS  through  the 
ITS  contract)  in  terms  of  software  licences,  a  smaller  terrain  map,  existing 
content,  and  modeling  and  coding  support  (through  a  sub-contract). 

5.  DRDC  Toronto  (C-IED  TDP)  is  a  key  collaborator  for  the  ITS  project;  both  in 
terms  of  sharing  VBS2  content  pertaining  to  C-IED,  and  identifying  future 
technology  transfer  opportunities  (e.g.,  with  the  CUTE  project). 
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5.1.2  Stakeholder  Analysis 

The  stakeholder  analysis  determined  that  the  implementation  plan  and  conduct  of  the 
project  must  be  cognisant  of  the  roles  of  stakeholders  during  the  different  project 
phases.  Areas  for  consideration  in  the  detailed  implementation  plan,  organized  by 
project  phase,  are  as  follows: 

•  Analysis  and  design  phase.  Identify  areas  where  capabilities  of  stakeholders  may 
provide  the  ITS  with  an  advantage.  During  the  analysis  and  design  phase,  the 
role  of  subject  matter  experts  is  particularly  important  and  demands  a  higher 
level  of  involvement  than  during  other  phases.  Communication  and  buy-in  for 
stakeholders  who  may  provide  subject  matter  knowledge  will  be  key  during  this 
phase. 

•  Implementation  phase.  Identify  areas  where  capabilities  of  stakeholders  may 
provide  the  ITS  with  an  advantage.  For  instance,  are  their  people  in  stakeholder 
organizations  with  the  skills  or  knowledge  to  do  reviews  or  contribute  media 
content  or  technologies  to  the  project? 

•  Evaluation  phase.  Identify  appropriate  stakeholders  to  participate  in  the 
evaluation  of  the  project  outcome,  and  stakeholders  who  can  create  a  larger 
degree  of  support  and  information  dissemination  about  the  results  of  the  ITS 
project. 

Finally,  it  is  recommended  that  the  project  team  maintains  a  ‘technology  watch’ 
throughout  the  duration  of  the  project.  The  technology  watch  will  include  activities 
pertaining  to: 

•  Monitoring  of  virtual  world  and  gaming  technologies; 

•  Monitoring  role  changes  in  the  stakeholder  community; 

•  Monitoring  for  new  stakeholders; 

•  Maintaining  stakeholder  involvement  at  the  right  time; 

•  Communicating  project  outcomes  with  stakeholders  and  receiving  any  new 
findings  or  knowledge  of  the  stakeholders  themselves; 

•  Providing  feedback  loops  as  a  way  to  keep  stakeholders  involved  in  the  project; 
and, 

•  Ensuring  that  project  deliverables  are  visible  to  the  appropriate  stakeholder  (to  be 
determined  on  a  case-by-case  basis). 
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5.2  Next  Steps 

It  is  important  to  note  again  that  the  scope  of  potential  ITS  developments  under 
consideration  might  likely  exceed  the  schedule  and  budget  available  to  develop  a 
prototype  and  deploy  the  prototype  into  the  existing  IEDD  course  schedule  to  perform 
experimentation.  As  such,  the  project  team  will  analyse  the  technology  and  instructional 
design  options  available  and  develop  a  detailed  implementation  plan  to  match  the 
development  of  the  IEDD  ITS  to  the  constraints  of  the  project  budget  and  schedule.  The 
work  plan  described  in  this  report  details  the  work  to  be  performed  to  produce  an  ITS 
prototype  to  support  the  evaluation  of  its  utility  as  a  component  of  the  existing  IEDD 
Operator  Course.  The  implementation  plan  and  schedule  will  continue  to  evolve  over 
the  next  phase  of  the  project  as  more  information  about  the  IEDD  Operator  Course, 
teaching  scenario  and  instructional  design  comes  to  light. 

The  report  recommends  a  number  of  tasks  to  be  conducted  early  during  the  next  phase 
of  the  project  in  order  to  provide  more  detailed  plans  for  the  implementation  of  ITS 
technologies  within  the  IEDD  Operator  Course: 

•  Questioning  technique  literature  review; 

•  Development  and  validation  of  the  IEDD  ITS  mission  scenario; 

•  Conduct  taxonomic  analysis  (including  instructional  design);  and, 

•  IEDD  ITS  development  (iteration  1). 

The  evaluation  plan  will  also  continue  to  evolve  in  light  of  the  findings  from  the 
taxonomic  analysis  and  the  development  of  the  functionality  and  capabilities  of  the 
IEDD  ITS  itself.  The  taxonomic  analysis  will  also  identify  and  describe  what  measures  of 
performance  and  the  tools  should  be  used  to  evaluate  the  efficacy  of  the  IEDD  ITS  to 
enhance  the  learning  experience  of  the  student.  Finally,  the  CAE  PS  team  will  contact 
the  Directorate  of  Military  Personnel  Operational  Research  and  Analysis  (DMPORA)  for 
ethical  approval  to  collect  physiological  data  from  CF  personnel  during  the  ITS 
evaluation  sessions.  In  addition,  the  CAE  PS  team  will  seek  guidance  from  the  ITS 
project’s  scientific  authority  regarding  the  requirement  for  ethical  approval  from  DRDC 
Toronto. 
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7  LIST  OF  ACRONYMS 


AAR 

After  Action  Review 

AFIILE 

Air  Force  Integrated  Information  and  Learning  Environment 

ALSC 

Army  Learning  Support  Centre 

ANA 

Afghan  National  Army 

ANP 

Afghan  National  Police 

ARP 

Applied  Research  Project 

CAMIX 

Civilian  Modeling  Experimentation 

CBT 

Computer  Based  Training 

CDA 

Canadian  Defence  Academy 

CF 

Canadian  Forces 

C-IED 

Counter  Improvised  Explosive  Device 

CUTE 

Counter  IED  Immersive  Training  Environment 

CO 

Commanding  Officer 

COTS 

Commercial-Off-the-Shelf 

DLR 

Directorate  of  Land  Requirements 

DLSE 

Directorate  of  Land  Synthetic  Environments 

DMPORA 

Directorate  of  Military  Personnel  Operational  Research  and 
Analysis 

DND 

Department  of  National  Defence 

DRDC 

Defence  Research  Development  Canada 

EDR 

Electrodermal  response 

EEG 

Electroencephalography 

EFIT 

Environmental  Familiarisation  and  Indicator  Trainer 

EW 

Electronic  Warfare 

FPS 

First  Person  Shooter 

GOTS 

Government  Off-the-shelf 

GSR 

Galvanic  Skin  Response 

GUI 

Graphical  User  Interface 

HCI 

Human  Computer  Interaction 

HF 

Human  Factors 

HRV 

Heart  Rate  Variability 

IAS 

Intelligent  Adaptive  System 

IED 

Improvised  Explosive  Device 

IEDD 

Improvised  Explosive  Device  Disposal 

ILS 

Index  of  Learning  Styles 

ITS 

Intelligent  Tutoring  System 

KS 

Knowledge  Sources 

KSA 

Knowledge,  Skills  and  Attitudes 
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LCMS 

LMS 

MTP 

NDHQ 

NPC 

NRC 

OMI 

OSC 

PGR 

P-Learn 

PO 

QRP 

QS 

QT 

RSP 

SA 

SCORM 

SCR 

SMAP 

SME 

TAM 

TDP 

UAV 

US 

RDECOM 

VBS2 


Learning  Content  Management  Systems 

Learning  Management  Systems 

Main  Teaching  Points 

National  Defence  Head  Quarters 

Non-Player  Characters 

National  Research  Council 

Operator  Machine  Interface 

On  Scene  commander 

Psychogalvanic  reflex 

Personal  learning  Environment 

Performance  Objectives 

Quick  Response  Personnel 

Qualification  Standards 

Questioning  Technique 

Render  Safe  Procedure 

Situation  Awareness 

Sharable  Content  Object  Reference  Model 
Skin  conductance  response 
Situation  Model  Argument  Map 
Subject  Matter  Expert 
Technology  Acceptance  Model 
Technology  Demonstration  Project 
Uninhabited  Air  Vehicle 

United  States  Research  and  Development  Engineering 
Command 

Virtual  Battle  Space  2 


-  103  — 


5045-1  Version  02 


28  May  2010 


©  Her  Majesty  the  Queen  as  represented  by  the  Minister  of  National  Defence,  2010 
©  Sa  majeste  la  reine,  representee  par  le  ministre  de  la  Defense  nationale,  2010 


CAE 


Stakeholder  Analysis  for  C-IED  Training  Courses 


APPENDIX  A  STAKEHOLDER  ANALYSIS  MEETINGS 

Table  7-1  describes  the  meetings  that  were  held  as  part  of  the  ITS  Stakeholder  Analysis. 

Table  7-1:  ITS  Stakeholder  Analysis  meetings 


ID# 

Visit 

Time 
and  Date 

Location 

Issues  Discussed 

CAE  PS 
Attendees 

1 

C-IED  Task  Force  /  Training,  NDHQ  (Maj 

Dan  Schurman) 

T:  613  992  2330 

M:  613  513  9104 

DAN. SCHURMAN@forces. gc.ca 

1100,  22nd 
January 

2010 

NDHQ,  Ottawa 

Content,  courses  taught,  training  needs  analysis.  Relationship 
with  other  IED  /  e-learning  initiatives  at  CFB  Gagetown. 

Emphasis  on  IEDD  Operator  Course. 

SB,  KO  and  CK 

2 

C-IED  Defeat  the  Device  Center  of 
Excellence,  CF  School  of  Military 

Engineering  (Capt  Luc-Frederic  Gilbert) 

T:  506  422  2000  ext.  7212 
Luc-Frederic.Gilbert  @  forces. gc.ca 

0900,  25th 
January 

2010 

Building  J-10 
(CFSME),  Room 
B209.CFB 

Gagetown, 

Fredricton  NB 

Tools,  approaches,  content,  courses  taught,  training  needs 
analysis.  Relationship  with  other  IED  /  e-learning  initiatives  at 

CFB  Gagetown. 

Emphasis  on  IEDD  Operator  Course. 

SB  and  KO 

3 

C-IED  Attack  the  Network,  Combat  Training 
Centre  (Maj  Andrew  Gimby) 

T:  506  422  2000  ext.  2654 

Andrew.Gimby@ 
forces. gc.ca 

1400,  25th 
January 

2010 

Tactics  School, 

CFB  Gagetown, 
Fredricton  NB 

Tools,  approaches,  content,  courses  taught,  training  needs 
analysis.  Relationship  with  other  IED  /  e-learning  initiatives  at 

CFB  Gagetown. 

SB  and  KO 

4 

Learning  and  Collaborative  Technologies 
Group,  National  Research  Council  of 
Canada,  Institute  for  Information 

Technology  (Dr  Rodrigue  Savoie) 

T:  506  861  0951 

Rod  rigue.Savoie@nrc-cnrc. gc.ca 

0930,  26th 
January 

2010 

NRC,  University  of 
New  Brunswick, 
Fredricton  NB 

Tools,  approaches.  Discuss  CDA  collaboration  and  relationship. 

Sub-contract  review  and  progress.  Mobile  learning  literature 
discussion. 

SB  and  KO 
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ID# 

Visit 

Time 
and  Date 

Location 

Issues  Discussed 

CAE  PS 
Attendees 

5 

Directorate  of  Land  Synthetic  Environments 
(Capt  Chris  Taff  and  Dr  Paul  Roman) 

CHRIS. TAFF@ 
forces. gc.ca 

T:  613  541  5010  ext  5042 

PAUL. ROMAN@forces. gc.ca 

1300,  II"1 
February 
2010 

Directorate  of  Land 
Synthetic 

Environments,  CFB 
Kingston  ON 

Content,  tools  used,  lessons  learnt.  Relationship  with  other  IED  / 
e-learning  initiatives  at  CFB  Gagetown. 

SB,  KO  and  CK 

5 

Directorate  of  Learning  Innovation, 

Canadian  Defence  Academy  (LCdr  Remi 
Tremblay) 

T:  613  541  5010  ext  3938 

Remi. Tremblay@forces. gc.ca 

0930,  12th 
February 
2010 

Canadian  Defence 
Academy,  CFB 
Kingston  ON 

Tools,  approaches,  and  lessons  learnt.  Discuss  CDA 
collaboration  and  relationship.  Relationship  with  other  IED  /  e- 
learning  initiatives  at  CFB  Gagetown  and  elsewhere. 

SB,  KO  and  CK 

6 

Counter-IED  TDP,  Collaborative 

Performance  and  Learning  Section,  DRDC 
Toronto  (Dr  Jerzy. Jarmasz  and  Sgt  Dorothy 
Wojtarowicz) 

T:  416  635  2000  ext  3081 

Jerzy.  Jarmasz@drdc-rddc.  gc.ca 

1100,  24th 
February 
2010 

DRDC  Toronto, 

1133  Sheppard  Ave 
West,  Toronto  ON 

Content,  tools  used,  lessons  learnt.  Relationship  with  other  IED  / 
e-learning  initiatives  at  CFB  Gagetown  and  elsewhere. 

SB  and  CK 
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APPENDIX  B  STAKEHOLDER  ANALYSIS  RESULTS 

Table  7-2  describes  the  results  of  the  ITS  Stakeholder  Analysis. 


Table  7-2:  ITS  Stakeholder  Analysis  results 


Stakeholder 

Organisation 

Stakeholder 

Name 

Role 

Stake  /  Interest 
in  Project 

ITS  Related 
Programs  and  Links 

ITS  Related 
Capabilities 

Map  to  Program 
Phase 

Stakeholder 

Needs 

Perceived 
Attitude  and 
Risks 

Project  Impact  on 
Stakeholder 

Connections  to  Other 
Stakeholders 

C-IED  Task  Force 
DND 

Ottawa,  ON 

Maj.  Dan 
Schurman 

NDHQ,  Ottawa, 

ON 

Training 

Development  Officer. 
Lead  on 

communications  with 
other  C-IED 
stakeholders. 

High  stake  and 
interest. 

High  influence. 
Ongoing  interest  in 
applications  of  project 
to  other  training 
needs. 

Involved  in  previous  DRDC 
studies. 

Current  project:  Analysis  of 
selection  process  of  IEDD 
operators. 

Access  to  Subject  Matter 
Experts. 

Access  to  training 
documentation. 

Analysis  and  Design. 

Implementation. 

Evaluation. 

Communication. 
Information  Provision. 
Team  member. 

Advice  and  Support. 

Highly  supportive. 

Key  stakeholder 
desiring  to  effect  change 
through  technologies. 

Low  Risk. 

High.  Will  provide  a 
tools/approach  that  can 
be  used  elsewhere  in 
the  Army  training 
requirements. 

Direct  Support:  Defeat  the  Device  Centre  of 
Excellence,  School  of  Mil.  Eng. 

Direct  Support:  C-IED  Attack  the  Network, 
Combat  Training  Centre. 

Direct:  DLR  is  part  of  C-IED  Task  Force. 
Desire  improved  connection  to  Army  Sim 
Centre. 

C-IED  Defeat  the 
Device  Centre  of 
Excellence 

School  of  Military 
Engineering 

IEED  Operator 
Course 

Gagetown,  NB 

Capt.  Gilbert 
Gagetown,  NB 

Point  of  Contact  for 
IEED  Operator 

Course. 

High  stake  and 
interest. 

High  influence. 
Ongoing  interest  in 
applications  of  project 
to  other  training 
needs. 

High  Power. 

High  Need. 

Linked  to  other  IED  courses 
and  training  requirements. 

Subject  Matter  Experts 
Training  design  and 
content  owner. 

Analysis  and  Design. 

Implementation. 

Evaluation. 

Communication. 
Information  Provision. 
Team  member. 

Advice  and  Support. 
Manage  expectations 
for  scope  and  timelines. 

Highly  supportive. 

Key  stakeholder 
interested  in  ensuring 
best  learning  experience 
for  all  students. 

Low  Risk. 

High  impact  for  team 
and  course 
responsibilities. 

Direct  Support:  C-IED  Task  Force. 

Direct  Support:  C-IED  Attack  the  Network, 
Combat  Training  Centre. 

Indirect:  Combat  Training  Centre  provides 
media  /  training  support  as  needed. 
Awareness:  DLR  is  part  of  C-IED  Task 

Force. 

Awareness:  DRDC  Toronto  C-IED  Projects. 

Combat  Training 
Centre 

Army  Training 
Gagetown,  NB 

Maj.  Gimby 
Gagetown,  NB 

Point  of  Contact  for 
Combat  Training 
Centre. 

Medium  stake  and 
interest. 

Ongoing  interest  in 
applications  of  project 
to  other  training 
needs. 

Low  Power. 

Low  Need. 

Linked  to  wide  range  of 
courses  and  training 
requirements. 

Knowledge  of  and  links 
to  other  courses. 
Knowledge  of  media 
produced  for  other 
courses. 

Analysis  and  Design. 

Implementation. 

Evaluation. 

Awareness  of  progress. 

Supportive. 

Low  impact. 

Direct  Support:  DRDC  (Toronto)  C-IED 
projects. 

Direct  Support:  Army  Learning  Support 
Centre  (ALSC) 

Indirect:  CF  School  off  Communiations  and 
Electronics  work  in  C-IED  and  RC  Defeat 
the  Device. 

Indirect:  School  of  Military  Intelligence 
(Kingston),  Tactical  Questioning  Course  (for 
intelligence  staff).  POC  Maj.  J.P.  Terfy. 
Indirect:  Defeat  the  Device  COE,  EW 
perspective. 

National  Rearch 
Council  (NRC) 

IIT  Branch 
Fredericton,  NB 

Rod  Savoie 
Moncton,  NB 

Group  Leader 

Learning  and 
Collaborative 
Technologies 

IIT,  NRC 

High  stake  and 
interest. 

CAE  PS 

Subcontractor. 

Parrallel  research 
interests  on  behalf 
Government  of 
Canada. 

Consulting  on  Personal 
Learning  Environments  to 
CDA. 

Cognitive  modelling  (Bruno 
Emund). 

Adaptive  Learnining 

System  (Doctor-Patient 
Example,  Bruce  Spencer). 

Subcontractor  for 
speciality  skills  in  Mobile 
Learning  and  Adaptive 
Learning. 

Analysis  and  Design. 

Implementation. 

Evaluation. 

Communication. 
Information  Provision. 
Team  member. 

Advice  and  Support. 

Highly  supportive. 

Key  stakeholder 
desiring  to  effect  change 
through  technologies. 

Low  Risk. 

Medium.  Achieves 
mandate  to  work  with 
other  government 
departments  and 
industry. 

Direct  Support:  Army  Learning  Support 
Centre  (ALSC). 

Direct  Support:  Various  project  with 

Canadian  Defence  Academy  (CDA). 
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Stakeholder  Analysis  for  C-IED  T raining  Courses 


Stakeholder 

Organisation 

Stakeholder 

Name 

Role 

Stake  /  Interest 
in  Project 

ITS  Related 
Programs  and  Links 

ITS  Related 
Capabilities 

Map  to  Program 
Phase 

Stakeholder 

Needs 

Perceived 
Attitude  and 
Risks 

Project  Impact  on 
Stakeholder 

Connections  to  Other 
Stakeholders 

Canadian  Defence 
Academy. 

Concept 

Development  and 
Exploration. 
Kingston,  ON 

LCdr.  (Navy) 

Remi  Tremblay 
Kingston,  ON 

Concept 

Development  and 
Exploration. 

Medium  stake  and 
interest. 

Ongoing  interest  in 
applications  of  project 
to  other  CF  wide 
training  needs. 

Medium  Power. 
Medium  Need. 

Work  in  Domains: 

1)  Future  learning  concepts 
and  architectures 
(PLEARN) 

2)  Simulation:  Virtual  World 
and  Gaming  exploration 
(CASE  Contract,  SIMON 
project) 

3)  Mobile:  Track  use, 
previous  lit  review. 

4)  Physical:  Biofeedback, 
Technology  Watch. 

5)  Cognitive  Task  Analysis: 
Visualized  decision  paths. 

Knowledge  of  and  links 
to  other  courses. 
Knowledge  of  media 
produced  for  other 
courses. 

Exploration  and  Lessons 
Learned  with  variety  of 
gaming  and  virtual  world 
technology. 

High  Level  Awareness: 
Analysis  and  Design. 
Implementation. 
Evaluation. 

Communication. 
Information  Provision. 
Team  member. 

Advice  and  Support. 

Highly  supportive. 

Key  stakeholder 
desiring  to  effect  change 
through  technologies. 

Low  Risk. 

Medium.  Achieves 
mandate  through  co¬ 
operation  and 
awareness. 

Direct  Support:  Army,  Navy,  Airforce 

Training  Organizations 

Direct  Support  and  Awareness:  DRDCs 
Awareness:  Adhoc  projects. 

Directorate  Land 

Synthetic 

Environment 

(DLSE) 

and 

Director  Land 
Requirements 
(DLR) 

Kingston,  ON 

POC:  Capt.  Chris 
Taff 

Second  POC:  Dr. 
Paul  Roman 

Technology  support 
for  Army  simulation 
and  training. 

Low  stake. 

High  interest. 

Ongoing  interest  in 
applications  of  project 
to  other  training 
needs. 

Use  of  some  common 
game  or  virtual  world 
technology. 

Developed  Ngrain  models 
for  300  mine  types. 

Deployed  on  mobile  device 
to  assist  in  recognition. 
Knowledge  of  pros  and 
cons  of  technologies. 
Knowledge  of  related 
programs. 

Limited  VBS  2 
development  capability 
through  subcontractors. 
Source  of  technology 
lessons  learned. 

Implementation 

Awareness  of  progress. 

Highly  supportive. 

High  Interest. 

Key  stakeholder 
desiring  to  effect  change 
through  technologies. 

Low  Risk. 

Low  impact. 

Direct:  Army  simulation  use  of  VBS  and 
other  technologies. 

Indirect:  Knowledge  of  many  other  CF 
programs. 

Indirect:  Links  to  Combat  Training  Centre 
and  ALSE  (Gagetown,  NB). 

DRDC  (Atlantic) 
HLA  Centre  of 
Excellence. 
Dartmouth,  NS. 

Brad  Dillman 
DRDC  (Atlantic) 
Software  Lead 
Dartmouth,  NS. 

Exploration  with 
game  and  virtual 
world  technology  to 
support  partners. 

Links  to  HLA  centre 
of  excellence. 

Medium  stake  and 
interest. 

Ongoing  interest  in 
applications  of  project 
to  other  DRDC 
initiatives. 

VBS  2  Modelling  of  Halifax 
harbour. 

VW  technology  to  simulate 
virtual  operations  room. 

Limited  VBS  2 
development  capability 
through  subcontractors. 
Source  of  lessons 
learned  for  Second  Life 
in  non-training 
applications. 

N/A 

Awareness  of  progress. 

Highly  supportive. 

High  Interest. 

Key  stakeholder 
desiring  to  effect  change 
through  technologies. 

Low  Risk. 

Low  impact. 

Direct  Support:  DRDC  (Toronto) 

DRDC 

IEDTDP 

Toronto,  ON 

Jerzy  Jamasz 
Toronto,  ON 

DRDC  Lead 

IED  Technology 
Demonstration 

Project 

Low  stake. 

High  interest. 

Ongoing  interest  in 
applications  of  project 
to  other  training 
needs. 

IED  TDP  has  some 
overlapping  stakeholders 
and  high  level  objectives. 

Domain  knowledge  and 
interest. 

Lessons  learned  in  VW 
technology. 

Analysis  and  Design. 

Implementation. 

Evaluation. 

Communication. 
Information  Provision. 
Advice  and  Support. 

Highly  supportive. 

High  Interest. 

Key  stakeholder 
desiring  to  effect  change 
through  technologies. 

Low  Risk. 

Low  impact. 

All. 

B-2 


28  May  2010 


©  Her  Majesty  the  Queen  as  represented  by  the  Minister  of  National  Defence,  2010 
©  Sa  majeste  la  reine,  representee  par  le  ministre  de  la  Defense  nationale,  2010 


5045-1  Version  02 


UNCLASSIFIED 


DOCUMENT  CONTROL  DATA 

(Security  classification  of  the  title,  body  of  abstract  and  indexing  annotation  must  be  entered  when  the  overall  document  is  classified) 


3.  TITLE  (The  complete  document  title  as  indicated  on  the  title  page.  Its  classification  is  indicated  by  the  appropriate  abbreviation  (S,  C,  R,  or  U)  in  parenthesis  at 
the  end  of  the  title) 

Stakeholder  Analysis  for  the  CF  Counter-IED  Training  Courses  (U) 

Analyse  des  intervenants  des  cours  d’instruction  des  FC  sur  la  neutralisation  des  IED 

(U) _ 

4.  AUTHORS  (First  name,  middle  initial  and  last  name.  If  military,  show  rank,  e.g.  Maj.  John  E.  Doe.) 

Simon  Banbury;  Kristine  Osgoode;  David  Unrau;  Chelsea  Kramer 

6a  NO.  OF  PAGES 

(Total  containing  information,  including 
Annexes,  Appendices,  etc.) 

119 

7.  DESCRIPTIVE  NOTES  (The  category  of  the  document,  e.g.  technical  report,  technical  note  or  memorandum.  If  appropriate,  enter  the  type  of  document, 
e.g.  interim,  progress,  summary,  annual  or  final.  Give  the  inclusive  dates  when  a  specific  reporting  period  is  covered.) 

Contract  Report 

8.  SPONSORING  ACTIVITY  (The  names  of  the  department  project  office  or  laboratory  sponsoring  the  research  and  development  -  include  address.) 

Sponsoring: 

Tasking:  DRDC  Toronto 


1 1 .  DOCUMENT  AVAILABILITY  (Any  limitations  on  the  dissemination  of  the  document,  other  than  those  imposed  by  security  classification.) 

Unlimited  distribution 

1  2.  DOCUMENT  ANNOUNCEMENT  (Any  limitation  to  the  bibliographic  announcement  of  this  document.  This  will  normally  correspond  to  the  Document 
Availability  (11),  However,  when  further  distribution  (beyond  the  audience  specified  in  (1 1)  is  possible,  a  wider  announcement  audience  may  be  selected.)) 

Unlimited  announcement 


9a.  PROJECT  OR  GRANT  NO.  (If  appropriate,  the  applicable 

research  and  development  project  or  grant  under  which  the  document  was 
written.  Please  specify  whether  project  or  grant.) 

14av 

10a.  ORIGINATOR'S  DOCUMENT  NUMBER  (The official 

document  number  by  which  the  document  is  identified  by  the  originating 
activity.  This  number  must  be  unique  to  this  document) 

DRDC  Toronto  CR  2010-059 


9b.  CONTRACT  NO.  (If  appropriate,  the  applicable  number  under  which 
the  document  was  written.) 

W771 1  -09-81 53/001  /TOR 


10b.  OTHER  DOCUMENT  NO(s).  (Any  other  numbers  under  which 
may  be  assigned  this  document  either  by  the  originator  or  by  the 
sponsor.) 


6b.  NO.  OF  REFS 

(Total  cited  in  document.) 


5.  DATE  OF  PUBLICATION 

(Month  and  year  of  publication  of  document.) 

June  2010 


2.  SECURITY  CLASSIFICATION 

(Overall  security  classification  of  the  document 
including  special  warning  terms  if  applicable.) 

UNCLASSIFIED 


1.  ORIGINATOR  (The  name  and  address  of  the  organization  preparing  the  document,  Organizations 
for  whom  the  document  was  prepared,  e.g.  Centre  sponsoring  a  contractor's  document,  or  tasking 
agency,  are  entered  in  section  8.) 

Publishing:  DRDC  Toronto 

Performing:  DRDC  CAE  Professional  Services,  1135  Innovation 

Toronto  Dr.,  Suite  3000 

Monitoring: 

Contracting: 


UNCLASSIFIED 


UNCLASSIFIED 


DOCUMENT  CONTROL  DATA 

(Security  classification  of  the  title,  body  of  abstract  and  indexing  annotation  must  be  entered  when  the  overall  document  is  classified) 

1 3.  ABSTRACT  (A  brief  and  factual  summary  of  the  document.  It  may  also  appear  elsewhere  in  the  body  of  the  document  itself.  It  is  highly  desirable  that  the  abstract 
of  classified  documents  be  unclassified.  Each  paragraph  of  the  abstract  shall  begin  with  an  indication  of  the  security  classification  of  the  information  in  the  paragraph 
(unless  the  document  itself  is  unclassified)  represented  as  (S),  (C),  (R),  or  (U).  It  is  not  necessary  to  include  here  abstracts  in  both  official  languages  unless  the  text  is 
bilingual.) 

(U)  This  document  presents  the  results  of  a  Stakeholder  Analysis  of  Canadian  Forces  (CF) 
Counter  Improvised  Explosive  Device  (IED)  training  courses.  The  Stakeholder  Analysis 
was  used  to  inform  the  design,  development  and  evaluation  of  an  Intelligent  Tutoring 
System  (ITS)  for  the  IED  Disposal  (IEDD)  Operator  Course.  This  work  was  conducted 
under  Contract  W771 1-09-81 53/001 /TOR  “Intelligent  Tutoring  for  Distance  Learning”  for 
Defence  Research  Development  Canada  (DRDC)  Toronto  by  CAE  Professional  Services 
(CAE  PS). 

One  of  the  most  challenging  activities  for  a  distance  education  facilitator  is  to  be 
responsive  to  student  needs  and  customize  the  learning  experience  to  the  student’s 
individual  learning  style,  help  socialize  the  student  to  the  institution,  and  keep  the  learner 
engaged  in  spite  of  the  isolated  environment.  To  achieve  these  goals,  Defence  Research 
and  Development  Canada  (DRDC)  -  Toronto  has  initiated  an  Applied  Research  Project 
(ARP)  to  investigate  mechanisms  to  improve  efficiency  and  effectiveness  for  the  Canadian 
Forces  (CF)  distance  learning  capability. 

In  order  to  support  these  goals,  a  Requirement  and  Stakeholder  Analysis  for  the  CF 
Counter-IED  Training  Courses  was  undertaken  by  CAE  PS  to  investigate  the 
requirements  of  stakeholders  of  the  CF  IEDD  Operator  Course  for  the  development  of 
adaptive  learning  technology  integration  and  validation  plans.  Given  the  high  failure  rate  in 
the  existing  IEDD  Operator  Course,  problematic  aspects  of  the  course  was  one  of  the  foci 
of  the  Stakeholder  Analysis.  The  discussion  with  stakeholders  focused,  in  part,  on  the 
feasibility  of  demonstrating  the  capability  of  intelligent  tutoring  technology  within  the  IEDD 
Operator  Course. 

The  report  presents  the  results  of  the  Stakeholder  Analysis  interviews  conducted  between 
the  22nd  January  2010  and  the  23rd  February  2010,  and  describes  the  recommendations 
for  implementing  adaptive  learning  and  intelligent  tutoring  technologies  into  a  suitable  CF 
distance-learning  course.  Following  this,  the  report  describes  the  implementation  plan  for 
the  IEDD  Operator  Course  ITS,  in  terms  of  its  content  and  capabilities,  and  describes  the 
notional  architecture  of  the  IEDD  ITS  in  more  detail,  as  well  as  the  schedule  for  the 
design,  implementation  and  evaluation  of  the  ITS.  Finally,  the  report  describes  the 
evaluation  plan  for  the  IEDD  Operator  Course  ITS  in  terms  of  the  methodology  and 
approximate  schedule  for  activities  pertaining  to  the  evaluation  of  the  ITS. 

The  report  also  recommends  a  number  of  tasks  to  be  conducted  early  during  the  next 
phase  of  the  project  in  order  to  provide  more  detailed  plans  for  the  implementation  of  ITS 
technologies  within  the  IEDD  Operator  Course: 

•  Questioning  technique  literature  review 

•  Development  and  validation  of  the  IEDD  ITS  mission  scenario 

•  Conduct  taxonomic  analysis  (including  instructional  design);  and, 

•  IEDD  ITS  development  (iteration  1). 

(U)  L’une  des  taches  les  plus  difficiles  qui  incombe  a  I’instructeur  a  distance  est  de  repondre 
aux  besoins  de  I’apprenant  et  de  personnaliser  I’acquisition  de  connaissances  en  fonction 
du  style  d’apprentissage  du  stagiaire,  d’entretenir  la  relation  entre  le  stagiaire  et 
I’etablissement  de  formation  et  de  conserver  la  motivation  de  I’apprenant  malgre  son 
isolement.  Pour  y  parvenir,  il  faut  examiner  des  moyens  d’ameliorer  la  rentabilite  et 
I’efficacite  de  la  capacite  d’apprentissage  a  distance  des  Forces  canadiennes  (FC).  Dans 
cette  optique,  on  a  entrepris  une  Analyse  des  besoins  et  des  intervenants  des  cours 
d’instruction  des  FC  sur  la  neutralisation  des  IED  afin  de  connaitre  les  besoins  des 
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